I am aware of the Targets.jl package that is currently in the General Registry. I will rename this package (taking suggestions), but had created this repository back in 2024 before there was a conflict and didn’t change the name when I picked up work on it a month ago.
If you came from R, there’s a chance that you may miss the {targets} package that keeps track of your analysis pipeline dependencies and only runs the sections that are out of date. I moved to Julia for the simulation speed, stayed for the ergonomics, but have always struggled to replicate the function-first workflow I had with R that didn’t rely on manually saving model results to files from scripts and hacking together some timestamp checks to determine whether they can be loaded or required a re-run.
A couple of years ago I started to sketch out some ideas of how to implement some of the functionality of the R package in a Julian way with multiple dispatch, but ultimately I didn’t get very far as I didn’t have the time to spend on it. Fast forward to the last couple of months, I thought I’d point Opus at it to see if I could get a functional prototype. I’m pleased to say that it’s currently more than sufficient for me to use in my work’s analysis pipelines where I really don’t want to have to re-run MCMC fits, but you should absolutely treat this as an alpha version if you wish to try it out. I haven’t come across any issues of silent failures (e.g., inappropriate code executions), but it’s certainly possible so you probably want to make sure your have good unit/integration tests of the results that are important to you.
The Motivation
As alluded to above, there are often occasions where you have long-running computations that you really don’t want to run more often than necessary. It would be great if there were a package that can tell what parts of your analysis pipeline are out of date (e.g., some of the upstream dependencies have changed) and only re-run those parts. Targets.jl (TBR) aims to accomplish this. It hashes the source code of the functions you write (it can also hash the lowered code, but this isn’t guaranteed to be stable across Julia versions so can cause false invalidations and unnecessary target re-runs), as well as the dependencies of that function invocation to create a graph of the analysis pipeline and handle these decisions for you. You can then use the tar_make() command to re-run everything that’s out of date, or, you can pass in specific target names to only re-run everything that’s out of date with respect to those targets (tar_make(;names = [:target_b, :target_d]).
How To Install
using Pkg; Pkg.add(url="https://github.com/arnold-c/Targets.jl")
Because this repository will be renamed, for the moment it can’t be installed from the General Registry and must be installed directly from GitHub. If you have thoughts for the name, please let me know!
How To Create And Run Pipelines
There are two ways you can create target pipelines; using the functional form or macros. Both the functional and macro forms should be roughly equivalent in terms of speed, but the macro form has to do a bit more work to track the dependencies. Regardless of the form you wish to use, the key takeaway is that don’t define very long pipelines - split them up (including across files that can be included by your main targets file) as this reduces the amount of code that needs to be lowered in a single pass so functions and macros that are used to read and load targets don’t have to do unnecessary work.
# _targets.jl
using Targets
using MyAnalysis # defines clean, fit_model, summarize, ...
@pipeline begin
@target raw_file = "data/raw.csv" format = :file
@target clean_data = clean(CSV.read(raw_file, DataFrame))
@target grid = [(alpha = a, depth = d) for a in (0.1, 0.5), d in 2:4] |> vec
@target fits = fit_model(clean_data, grid) pattern = grid
@target summary = summarize(fits)
end
Once you have defined a pipeline you can run it from the REPL, check on what’s outdated, and visualize the graph of dependencies (colour-coded by status. If you use GLMakie or WGLMakie you can select certain nodes to show/hide to filter down the view (you can also specify names = [:target_a, :target_c], as with tar_make()), as well as expanding the selection to include all upstream and downstream dependencies.
# In the REPL
using Targets
tar_make() # builds everything
tar_make() # builds nothing: everything is up to date
tar_read(:summary) # the stored value
@tar_load clean_data fits # clean_data = tar_read(:clean_data), ...
tar_outdated() # what would rebuild, and why
tar_list() # target names by section, in build order
tar_status() # the same, with each target's status, in colour
using WGLMakie
tar_visnetwork() # WGLMakie graph visualization of the pipeline, coloured by status, that can be manipulated
Targets sets the seed for each computation so each re-run will produce the same results, regardless of the computation ordering, until you update the target function or its seed. This can be set from within the tar_pipeline(;seed=1234)/@pipeline seed = 123 begin ... end setup, or via LocalPreferences.toml file and Preferences.jl.
In addition to creating simple pipelines that are manually named, you can map over parameters using static and dynamic branching. This allows you to apply the same set of functions to numerous sets of parameters. The static branching is when you know the exact parameters you wish to map over (with associated names).
# Static branching: `shifted_low`, `shift_summary_low`, ... one pair per named shift.
# Within a branch, `shifted` is that branch's `shifted` target.
@map((low = 1, high = 2)) do k
@target shifted = cleaned .+ k
@target shift_summary = summarize(shifted)
end
# Outside the @map, `shift_summary` is all of its branches: a NamedTuple keyed
# by suffix, `(low = ..., high = ...)`.
@target shifted_total = sum(s.total for s in shift_summary)
Dynamic branching allows you to pass in an object that may change in size and map your function(s) over that set of (undefined) parameters using the pattern kwarg. With dynamic branching, the parameters are hashed and used in the dynamically generated name.
score(alpha, beta) = alpha * beta
@pipeline begin
@target alphas = [0.1, 0.5]
@target betas = [1, 2]
# every combination: (0.1, 1), (0.1, 2), (0.5, 1), (0.5, 2)
@target every_pair = (alpha = alphas, beta = betas, value = score(alphas, betas)) pattern = cross(alphas, betas)
# in step, element i with element i: (0.1, 1), (0.5, 2)
@target in_step = (alpha = alphas, beta = betas, value = score(alphas, betas)) pattern = map(alphas, betas)
# or build the combinations yourself and branch over one target
@target grid = [(alpha = a, beta = b) for a in alphas for b in betas]
@target over_grid = (grid..., value = score(grid.alpha, grid.beta)) pattern = grid
end
Finally, it’s possible to include Quarto documents in your pipeline, so you can make sure your documents are using up-to-date objects in the rendered version you read. The @quarto/tar_quarto() macro/function looks for {julia} blocks and instances of tar_read()/tar_load() within them, as well as recursively searching included files (include("file.jl")) within these blocks to look for targets loaded from a regular Julia file (or via quarto include shortcodes {{< include file.qmd >}}).
There are a couple of other things that can be set manually or with a LocalPreferences.toml file and Preferences.jl, such as the number of tasks you wish to run simultaneously, or if you wish to use distributed workers instead of multithreading, but I think that this announcement is getting long enough.
If you use it and find it useful, or something that’s holding you back from using it/annoyance, please let me know as I’d like to get some feedback before I register the package.