Targets.jl (To Be Renamed) - Keep Your Analysis Pipelines Up-To-Date

I am aware of the Targets.jl package that is currently in the General Registry. I will rename this package (taking suggestions), but had created this repository back in 2024 before there was a conflict and didn’t change the name when I picked up work on it a month ago.

If you came from R, there’s a chance that you may miss the {targets} package that keeps track of your analysis pipeline dependencies and only runs the sections that are out of date. I moved to Julia for the simulation speed, stayed for the ergonomics, but have always struggled to replicate the function-first workflow I had with R that didn’t rely on manually saving model results to files from scripts and hacking together some timestamp checks to determine whether they can be loaded or required a re-run.

A couple of years ago I started to sketch out some ideas of how to implement some of the functionality of the R package in a Julian way with multiple dispatch, but ultimately I didn’t get very far as I didn’t have the time to spend on it. Fast forward to the last couple of months, I thought I’d point Opus at it to see if I could get a functional prototype. I’m pleased to say that it’s currently more than sufficient for me to use in my work’s analysis pipelines where I really don’t want to have to re-run MCMC fits, but you should absolutely treat this as an alpha version if you wish to try it out. I haven’t come across any issues of silent failures (e.g., inappropriate code executions), but it’s certainly possible so you probably want to make sure your have good unit/integration tests of the results that are important to you.

The Motivation

As alluded to above, there are often occasions where you have long-running computations that you really don’t want to run more often than necessary. It would be great if there were a package that can tell what parts of your analysis pipeline are out of date (e.g., some of the upstream dependencies have changed) and only re-run those parts. Targets.jl (TBR) aims to accomplish this. It hashes the source code of the functions you write (it can also hash the lowered code, but this isn’t guaranteed to be stable across Julia versions so can cause false invalidations and unnecessary target re-runs), as well as the dependencies of that function invocation to create a graph of the analysis pipeline and handle these decisions for you. You can then use the tar_make() command to re-run everything that’s out of date, or, you can pass in specific target names to only re-run everything that’s out of date with respect to those targets (tar_make(;names = [:target_b, :target_d]).

How To Install

using Pkg; Pkg.add(url="https://github.com/arnold-c/Targets.jl")

Because this repository will be renamed, for the moment it can’t be installed from the General Registry and must be installed directly from GitHub. If you have thoughts for the name, please let me know!

How To Create And Run Pipelines

There are two ways you can create target pipelines; using the functional form or macros. Both the functional and macro forms should be roughly equivalent in terms of speed, but the macro form has to do a bit more work to track the dependencies. Regardless of the form you wish to use, the key takeaway is that don’t define very long pipelines - split them up (including across files that can be included by your main targets file) as this reduces the amount of code that needs to be lowered in a single pass so functions and macros that are used to read and load targets don’t have to do unnecessary work.

# _targets.jl
using Targets
using MyAnalysis            # defines clean, fit_model, summarize, ...

@pipeline begin
    @target raw_file   = "data/raw.csv"            format = :file
    @target clean_data = clean(CSV.read(raw_file, DataFrame))
    @target grid       = [(alpha = a, depth = d) for a in (0.1, 0.5), d in 2:4] |> vec
    @target fits       = fit_model(clean_data, grid)    pattern = grid
    @target summary    = summarize(fits)
end

Once you have defined a pipeline you can run it from the REPL, check on what’s outdated, and visualize the graph of dependencies (colour-coded by status. If you use GLMakie or WGLMakie you can select certain nodes to show/hide to filter down the view (you can also specify names = [:target_a, :target_c], as with tar_make()), as well as expanding the selection to include all upstream and downstream dependencies.

# In the REPL
using Targets
tar_make()                    # builds everything
tar_make()                    # builds nothing: everything is up to date
tar_read(:summary)            # the stored value
@tar_load clean_data fits     # clean_data = tar_read(:clean_data), ...
tar_outdated()                # what would rebuild, and why
tar_list()                    # target names by section, in build order
tar_status()                  # the same, with each target's status, in colour

using WGLMakie
tar_visnetwork()              # WGLMakie graph visualization of the pipeline, coloured by status, that can be manipulated

Targets sets the seed for each computation so each re-run will produce the same results, regardless of the computation ordering, until you update the target function or its seed. This can be set from within the tar_pipeline(;seed=1234)/@pipeline seed = 123 begin ... end setup, or via LocalPreferences.toml file and Preferences.jl.

In addition to creating simple pipelines that are manually named, you can map over parameters using static and dynamic branching. This allows you to apply the same set of functions to numerous sets of parameters. The static branching is when you know the exact parameters you wish to map over (with associated names).

# Static branching: `shifted_low`, `shift_summary_low`, ... one pair per named shift.
# Within a branch, `shifted` is that branch's `shifted` target.
@map((low = 1, high = 2)) do k
    @target shifted       = cleaned .+ k
    @target shift_summary = summarize(shifted)
end

# Outside the @map, `shift_summary` is all of its branches: a NamedTuple keyed
# by suffix, `(low = ..., high = ...)`.
@target shifted_total = sum(s.total for s in shift_summary)

Dynamic branching allows you to pass in an object that may change in size and map your function(s) over that set of (undefined) parameters using the pattern kwarg. With dynamic branching, the parameters are hashed and used in the dynamically generated name.

score(alpha, beta) = alpha * beta
@pipeline begin
    @target alphas = [0.1, 0.5]
    @target betas  = [1, 2]

    # every combination: (0.1, 1), (0.1, 2), (0.5, 1), (0.5, 2)
    @target every_pair = (alpha = alphas, beta = betas, value = score(alphas, betas))  pattern = cross(alphas, betas)

    # in step, element i with element i: (0.1, 1), (0.5, 2)
    @target in_step = (alpha = alphas, beta = betas, value = score(alphas, betas))  pattern = map(alphas, betas)

    # or build the combinations yourself and branch over one target
    @target grid = [(alpha = a, beta = b) for a in alphas for b in betas]
    @target over_grid = (grid..., value = score(grid.alpha, grid.beta))  pattern = grid
end

Finally, it’s possible to include Quarto documents in your pipeline, so you can make sure your documents are using up-to-date objects in the rendered version you read. The @quarto/tar_quarto() macro/function looks for {julia} blocks and instances of tar_read()/tar_load() within them, as well as recursively searching included files (include("file.jl")) within these blocks to look for targets loaded from a regular Julia file (or via quarto include shortcodes {{< include file.qmd >}}).

There are a couple of other things that can be set manually or with a LocalPreferences.toml file and Preferences.jl, such as the number of tasks you wish to run simultaneously, or if you wish to use distributed workers instead of multithreading, but I think that this announcement is getting long enough.

If you use it and find it useful, or something that’s holding you back from using it/annoyance, please let me know as I’d like to get some feedback before I register the package.

It looks like a good complement for DrWatson.jl. A quick suggestion for fun, you could call it Gladstone.jl.
Otherwise, very useful tool !

Interesting stuff! Is this the substitute for having to comment out lines in a script? Is it correct that if there are changes in early stages of the pipeline, everything is called again, whereas if the changes are in the end, then only the last parts are called?
Also, how does it differ from Makeitso.jl?

Thanks! And RE Glastone.jl that’s a pretty fun name, but particularly given I don’t have any affiliation with the DrWatson.jl project, I think I would prefer the name to be a little bit more descriptive of the package’s purpose/homage to the R targets package that it draws more than a little inspiration from.

Effectively a substitute for having to manually use if-else statements that check against saved or cached results by name and timestamp (and saving the various parameters within the jld2 file that are checked).

Yes, that is correct - making a change to the code early in the pipeline (including updating an argument of a function call) will re-run all the later sections that rely on its result. If you only update later targets, then only their downstream dependencies are re-run.

In many ways it is similar to Makeitso.jl. If you find that fits your needs then by all means please continue with it as it’ll certainly be more stable than mine. For me I ultimately didn’t mesh with the ergonomics of it, and I didn’t love the DataFrame-centric approach. Coming from R I found the approach outlined in targets more intuitive where all the targets are specified in a pipeline with their arguments as you’d call them in the REPL (or as close to as possible), with a single make call to actually run the pipeline. In Makeitso.jl your package functions have to be wrapped in a @target and variables passed via kwargs, rather than just as you would call the function. A consequence of this is, as far as I can tell, Makeitso.jl doesn’t track the code defined outside of @target or @sweep calls, so changes to your package function methods wouldn’t trigger a recomputation. Targets.jl does track everything passed into the pipeline so should be more sensitive to your code changes.

From Makeitso.jl’s README: “The recipe supplied to @target and @sweep are hashed and this hash is used to track any changes done to the code, which would invalidate the computed values and all values that depend on them. This system can of course be fooled easily by having the recipes rely on untracked code”

I also wanted a more clearly defined definition of a pipeline - Makeitso.jl will make anything that is defined as a target anywhere in the file you run from the REPL. I wanted everything to have to be specified in the pipeline block. This allows me to specify multiple pipelines and then only run the one I’m interested in, without necessarily needing to split them up across files and worry about running make() on each target manually. This is related to another big difference - with Makeitso.jl it’s possible to have multiple targets with the same name, just with different arguments. I wanted there to be a clear distinction and only allow for a target name to be used once in the pipeline.

Other things that are different: I’ve added functions like tar_outdated()/tar_status() to show the state of your pipeline; tar_visnetwork()/tar_dot() to visualize your pipeline relationships; I allow you to run targets concurrently, or using Distributed.jl workers; I have Quarto targets for literate programming; I have added the tar_read()/tar_load() functions so you can load any object from a completely different script/file that may or may not be in the analysis pipeline (e.g., Quarto document).

It’s been a while since I’ve used Makeitso.jl, so there is a chance that my memory and re-reading the source code has lead me to a false conclusion (apologies if that’s the case). But it seems far more targeted at ensuring you have specific computation combinations run and up-to-date (and is a great complement to DrWatson.jl), and Targets.jl is focussed on the whole analysis pipeline, so it is broader in scope.