# Exploratory research project workflow

**URL:** <https://discourse.julialang.org/t/exploratory-research-project-workflow/53030>\
**Category:** General Usage\
**Created:** [January 8, 2021, 8:21am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030 "2021-01-08T08:21:05Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [January 9, 2021, 2:10pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/21 "2021-01-09T14:10:25Z")

</div>

I find it easy to use because it’s built on top of git and very agnostic to your programming language, tools, etc. If you are currently running scripts which read some files and write to others, and that chain of computations can be described by a DAG, then it will work for you.

How you structure experiments (by different git branches, or by commits, or just in the file structure) is up to you, although they provide some tooling for tracking metrics and comparing different branches/experiments. 90% of use cases will be covered just by pull/push/add/run/repro, kind of like how 90% of version control needs are covered by a few git commands.

It contrasts to other ML pipeline tools because it is less opinionated and less tied to a particular workflow/programming language. This was the case when I surveyed the options about 1 year ago, and others may have developed since then. I think some people may even use dvc in conjuction with other workflow tools, because it is relatively non-intrusive.

I don’t find there to be any strongly negative aspects to it, but it does keep you honest about any dependencies you mark and changes to those dependencies. Some people also find the one experiment/one branch idea to be too heavy, but I find that if you’re precise about what an “experiment” is then it’s not a problem.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 9, 2021, 8:50pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/22 "2021-01-09T20:50:19Z")

</div>

Indeed, `dvc` looks great and easy to use.  
I read the documentation and watched the tutorials yesterday. And I noted a few potential issues, but there might be a simple solution to it:

- not clear how to deal with advanced plots produced as results of the script, should one treat them as models and data tracked by dvc?
- I had an impression that dvs assumes one pipeline per project. Wondering how one deals with several ones (separate folders with `dvc init`? how to navigate in between?)
- it seems that `repro` and the pipeline requires starting Julia for every step. Sometimes with Julia, it is good to keep running the session.

@platawiec, could you please share your experience?

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 9, 2021, 10:03pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/23 "2021-01-09T22:03:56Z")

</div>

Sorry, some of the questions were too simple

> [@misha\_mikhasenko](#):
>
> - not clear how to deal with advanced plots produced as results of the script, should one treat them as models and data tracked by dvc?

An idea in CML is to reproduce them in the push pipeline

> [@misha\_mikhasenko](#):
>
> I had an impression that dvs assumes one pipeline per project. Wondering how one deals with several ones (separate folders with `dvc init` ? how to navigate in between?)

`dvc repro` takes the name of the pipeline as an agrument, so it is all scalable

I would love to see some examples already working with `julia`. Is there anything public?

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 10, 2021, 11:26am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/24 "2021-01-10T11:26:21Z")

</div>

If you have time, could you also give a bit of a high-level description?

I’m looking at a tutorial too and it seems like it’s just Git but with metadata files pointing to the data. Then, I don’t understand why the DAG is needed.

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 10, 2021, 11:37am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/25 "2021-01-10T11:37:14Z")

</div>

> [@rikh](#):
>
> Then, I don’t understand why the DAG is needed.

Ah, because DVC assumes the files to be too big to diff, so you cannot merge branches back into the main branch.

---

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [January 10, 2021, 3:30pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/26 "2021-01-10T15:30:49Z")

</div>

> [@misha\_mikhasenko](#):
>
> it seems that `repro` and the pipeline requires starting Julia for every step. Sometimes with Julia, it is good to keep running the session.

Yes, this is more of a script-based workflow for long computations that need to be tracked as part of a report or other deliverable. For rapid prototyping I use a Pluto notebook to try out new ideas/plots etc, then migrate it into the dvc pipeline as it finalizes.

Something like [GitHub - dmolina/DaemonMode.jl: Client-Daemon workflow to run faster scripts in Julia](https://github.com/dmolina/DaemonMode.jl) would probably work to maintain a Julia session across runs within this framework, though I haven’t tried it myself.

> [@rikh](#):
>
> I’m looking at a tutorial too and it seems like it’s just Git but with metadata files pointing to the data. Then, I don’t understand why the DAG is needed.

When I say “DAG”, I’m referring to the computational pipeline that is the user specifies. A common pattern I follow is something like:

```julia
[data/initial-data]
-> 
(data_processing_script.jl)
->
[data/processed-data]
-> 
(data_plotting_script.jl)
->
[data/plots]

```

Where `[]` is denoting some folder containing files (input/output) and `()` is denoting a computational step. In general a script may input from multiple stages, but the pipeline always satisfies the properties of a directed acyclic graph (DAG). In this case, we just have a two-stage computation. Now, if I change the code in `data_plotting_script.jl`, dvc will recognize that the code is different and re-run _only that stage_. In contrast, if I change `data_processing_script.jl` or the contents of the `data/initial-data` folder, then it will re-run the chain up to the point where I made a change.

There is some overhead to this process! I need to specify the stages, make sure the scripts run, pay the julia start-up time cost, and think about the staging and logical boundaries of the computation. That’s why (as I mentioned above) I usually prototype it out in a Pluto notebook or through a test-driven development process in my IDE. But, if you just need it to store your data with your git commit, I find it works for that too.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 11, 2021, 5:35pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/27 "2021-01-11T17:35:25Z")

</div>

How do you deal with the config files?

- TOML, Yaml, or julia scripts?
- Several nested or a single with `const` hyperparameters?

[Previous page](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030.md?page=1)