# Workfow for data analysis project: avoid re-running with many inputs, intermediate steps, and models
**URL:** https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997
**Category:** New to Julia
**Tags:** workflow
**Created:** [October 8, 2020, 6:34pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997 "2020-10-08T18:34:58Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@jdossgollin](https://discourse.julialang.org/u/jdossgollin)
#### Post date: [October 8, 2020, 6:34pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/1 "2020-10-08T18:34:58Z")
A typical data analysis project, for me, has many steps. For example, I might read in some datasets, fit some probabilistic models, create a database of simulations from this posterior model, and optimize something over that database of simulations. Then I might create plots of the raw data, the posterior simulations, and the optimized quantities. Each step in this workflow can be computationally intense, and so I’d like to avoid needing to re-run steps whenever possible.
In previous projects, I’ve used multi-language scripts with a workflow manager like Snakemake to keep my data analysis project in a straightforward workflow. This scripting workflow works well. However, I understand that this isn’t the best way to work with Julia (although there are hacks to make Julia sort of work like a scripting language, it’s not what it was designed for).
What is the best practice to keep track of dependencies in multi-step analysis workflows? the options that I’m aware of are
- use Julia with Snakemake and eat the startup/load time
- put everything in a `main.jl` file and input a bunch of files, commenting out the ones that don’t need to be re-run
- try to cache things
Is there a better way? Many thanks!
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [October 8, 2020, 6:49pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/2 "2020-10-08T18:49:38Z")
use a notebook?
---
### Author:  [@jdossgollin](https://discourse.julialang.org/u/jdossgollin)
#### Post date: [October 8, 2020, 6:52pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/3 "2020-10-08T18:52:49Z")
A notebook is great for the last step – take all the outputs and make plots. A notebook doesn’t do anything about dependency graphs (I update something in the middle and want to propagate changes through without re-running the previous steps)…
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [October 8, 2020, 6:54pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/4 "2020-10-08T18:54:13Z")
[https://github.com/fonsp/Pluto.jl](https://github.com/fonsp/Pluto.jl) tracks cells dependency and auto re-run
---
### Author:  [@jdossgollin](https://discourse.julialang.org/u/jdossgollin)
#### Post date: [October 8, 2020, 6:55pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/5 "2020-10-08T18:55:52Z")
Yeah I saw the juliacon presentation – really cool. But seems like this would just work in a single session, rather than over a few weeks/months while I’m working (possibly collaboratively) on a project?
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [October 8, 2020, 7:00pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/6 "2020-10-08T19:00:57Z")
> [@jdossgollin](#):
>
> a few weeks/months while I’m working (possibly collaboratively) on a project
looks like your intermediate results need to be written to disks anyways, then maybe use DrWatson.jl workflow or something.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [October 8, 2020, 7:02pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/7 "2020-10-08T19:02:36Z")
You probably want something akin to [DrWatson.jl](https://github.com/JuliaDynamics/DrWatson.jl). It provides a structure for this kind of workflow.
Also, please don’t comment and uncomment lines as needed! Use global Booleans to control this kind of behavior.
Another trick I like is to keep many data-sets in a dictionary, and use `@pack` and `@unpack` to work with them inside functions.
---
### Author:  [@jdossgollin](https://discourse.julialang.org/u/jdossgollin)
#### Post date: [October 8, 2020, 7:14pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/8 "2020-10-08T19:14:13Z")
> You probably want something akin to [DrWatson.jl](https://github.com/JuliaDynamics/DrWatson.jl). It provides a structure for this kind of workflow.
This looks like what I was looking for! I’ll check it out in more detail.
> please don’t comment and uncomment lines as needed! Use global Booleans to control this kind of behavior.
This was meant to be an example of what I don’t want to do! 🙂
> Another trick I like is to keep many data-sets in a dictionary, and use `@pack` and `@unpack` to work with them inside functions.
Are there performance benefits/costs to putting datasets in a dictionary vs. creating a `struct` for them?
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [October 8, 2020, 7:21pm UTC](https://discourse.julialang.org/t/workfow-for-data-analysis-project-avoid-re-running-with-many-inputs-intermediate-steps-and-models/47997/9 "2020-10-08T19:21:56Z")
No, as long as there is a function barrier before the actual computation, you can put something in a type-unstable object and then everything will infer just fine when you take it out of the object to do the computation.