# Using GNU Make, DrWatson, and Julia scripts to track dependencies and automate script running

**URL:** <https://discourse.julialang.org/t/using-gnu-make-drwatson-and-julia-scripts-to-track-dependencies-and-automate-script-running/103998>\
**Category:** General Usage\
**Tags:** question, drwatson, pipelines\
**Created:** [September 18, 2023, 6:38pm UTC](https://discourse.julialang.org/t/using-gnu-make-drwatson-and-julia-scripts-to-track-dependencies-and-automate-script-running/103998 "2023-09-18T18:38:41Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![arnold-c](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arnold-c/32/222741_2.png) [@arnold-c](https://discourse.julialang.org/u/arnold-c)\
**Post date:** [September 18, 2023, 6:38pm UTC](https://discourse.julialang.org/t/using-gnu-make-drwatson-and-julia-scripts-to-track-dependencies-and-automate-script-running/103998/1 "2023-09-18T18:38:41Z")

</div>

I have a number of scripts that I am trying to run using GNU Make, but within them, I am using DrWatson’s `@produce_or_load` macro to save parameter combinations to different files. I would like to use GNU Make to track dependencies and re-run later scripts when ones higher in the dependency tree have been updated. My understanding of Make is that it requires the output of a Make command to be a file so it can compare file modification times to calculate what needs to be re-run. If I use DrWatson to save the outputs instead of Make, is there a good way to have Make track dependencies? Currently I am using `touch` to create an empty temporary file with the same name as the script file, but this feels a bit hacky, and doesn’t track the actual outputs. It also seems to make it hard to use Make automatic variables etc (although, I am very new to Make, so I could just be missing something).

In the below mock example, I’m just using JLD2 to directly save (and load) the output, but it’s the same concept as using `@produce_or_load` in more complicated situations later - the saving and loading happens within the Julia scripts.

**Makefile**

```julia
.PHONY: all
all: tmp/single-sim_setup tmp/single-sim

tmp/single-sim_setup: src/single-sim_setup.jl
	julia $^
	@touch $@

tmp/single-sim: src/single-sim.jl tmp/single-sim_setup
	julia $^
	@touch $@

.PHONY: clean
clean:
    rm -rf tmp/*
    rm -rf data/singlesim/*

```

**src/single-sim\_setup.jl**

```julia
using DrWatson
@quickactivate "TestCode"

using JLD2

states = [100, 200, 300]
time = 0:1:100

jldsave("data/singlesim/single-sim_setup.jld2"; states, time)

```

**scripts/single-sim.jl**

```julia
#%%
using DrWatson
@quickactivate "TestCode"

using UnPack
using JLD2
using Distributions
using DataFrames

#%%
@unpack states, time = load("data/singlesim/single-sim_setup.jld2")

#%%
out = map(mean -> rand(Normal(mean), length(time), states)
out_df = DataFrame(out, :auto)

#%%
jldsave("data/singlesim/single-sim_arrays.jld2"; out, out_df)

```

I originally used lots of chained `include()` statements between scripts, but that made it difficult to develop multiple scripts that depended on the same source data, but had different requirements and tasks. I also wanted to make sure that everything was re-run when the dependencies were updated, and it was getting hard to keep track of which scripts to re-run.

If there is a better way of doing this (either with or without Make), please let me know. And if you need more information, I would be happy to expand upon anything.