# Exploratory research project workflow

**URL:** <https://discourse.julialang.org/t/exploratory-research-project-workflow/53030>\
**Category:** General Usage\
**Created:** [January 8, 2021, 8:21am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030 "2021-01-08T08:21:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 8:21am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/1 "2021-01-08T08:21:05Z")

</div>

People scale academic research, what is your workflow in Julia?  
I mean,

- how do you develop your project,
- in which point move functions to the module,
- when writing tests,
- how you scale,
- how you pass parameters between different parts,
- how do you save an intermediate representation?

Recently I watched a [great talk by Gaël Varoquaux](https://www.youtube.com/watch?v=AaqsGRKdoQ0&list=LL&index=7&ab_channel=SwissPythonSummit) at Python Summit. I was surprised, that his approach was similar to what I do eventually after years of struggling. It seems that the problem of the evolving code is common in many fields. I am curious to hear how people deal with it.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 8:24am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/2 "2021-01-08T08:24:59Z")

</div>

Find a similar post

> [@Workflow tips for small-team academic projects](https://discourse.julialang.org/t/workflow-tips-for-small-team-academic-projects/40965):
>
> I’m a longtime MATLAB user and have been learning Julia the past few weeks with a view to switch at least computationally intensive, if not all, coding work. But while I think I’ve gotten the hang of language basics like syntax, types, multiple dispatch, etc., I’m struggling to develop a comfortable workflow, or to see how it would scale once I bring collaborators on-board. I’ve read tips from Julia veterans scatted around this forum and elsewhere, but they seem to mainly be geared at people de…

with a lot of useful info

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 8:33am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/3 "2021-01-08T08:33:14Z")

</div>

Let’s narrow the scope. There are things that are a must:

- interactive environment with inline evaluation (VScode)
- version control. Git
- heavy data files stored separately
- a lot of plots

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [January 8, 2021, 10:08am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/4 "2021-01-08T10:08:46Z")

</div>

What I usually do is to create a folder for my new project, then a file in there and start coding, load the data, make some plots, write some functions, etc. Then if my file is getting big, I want to reuse code in another file, and my functions don’t change much anymore I create a “lib” folder and move my functions in files that I include in my scripts. Usually I also have a “plot” folder for plots, a “data” one for saving stuff (CSV and JLD2 mostly) and a “config” one for parameters and constant data. For tests at that point I sometimes just use some `@assert` in my files and look at the plots.

Then if I figure out that some of the code I wrote will be useful for other projects in the future and contains a coherent set of concepts, I move it to a package and clean it up, but that doesn’t happen very often.

Essentially I start in the most lazy way possible and make things more solid only when needed. That said it depends a bit on the project, that workflow is really for exploratory, one-shot analysis. If I have a good foreview on where the project will go I might be more organised. I usually git the folder but not always include a Project file (more by laziness than anything).

---

<div class="post-metadata">

**Author:** ![Jakob](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jakob](https://discourse.julialang.org/u/Jakob)\
**Post date:** [January 8, 2021, 2:45pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/5 "2021-01-08T14:45:43Z")

</div>

You might be interested in [DrWatson.jl](https://github.com/JuliaDynamics/DrWatson.jl) which basically automates this setup process and has some utility functions for navigating and activating a project.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 4:57pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/6 "2021-01-08T16:57:47Z")

</div>

I see, great, it also looks similar to what I do, however, I always create a module to have the namespace and a nice way to load all function and structure folders (`src`, `test`, `scripts`).  
How do you load your library without a module, by `include`?

The other thing,  
I am currently struggling to find a way to run the whole analysis that is a chain of different scripts with intermediate outputs. Any experience?

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 5:02pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/7 "2021-01-08T17:02:00Z")

</div>

`DrWatson` helped me realizing how important it is to `activate` local environment before running the code. Now, `vscode` does it automatically, so I stopped using it. For generation of packages, I use `PkgTemplates` that also generates settings for the GitHub CI.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 8, 2021, 5:06pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/8 "2021-01-08T17:06:13Z")

</div>

> [@misha\_mikhasenko](#):
>
> How do you load your library without module, by `include` ?

An exploratory workflow with a `main.jl` file which calls different scripts with `include` is a very good / acceptable workflow.

If you are only working with a single file then you want to use Revise.jl’s `includet`, which means you dont have to run `include` all the time.

But you should really be trying to put everything in functions.

Here’s what I do.

1. Start out with `main()` at the bottom of your script.
2. Use `includet` on that file and call `main()` when you want to run everywhint
3. Write everything in `main`, when it gets too big, break things into smaller functions that are called within `main`
4. Use `Exfiltrator.jl` extensively in debugging, as `@exfiltrate; error("")` when you want to break and send everything in that function’s local scope to global scope for inspection at the REPL
5. Use dictionaries, named tuples, and custom structs to store information. Pass large objects to the functions and use `@unpack` to work with just the things you need. This means you don’t have to agonize about what you are and are not passing to your sub-functions.
6. Just run `main()` all the time. If you use `includet` then everything will always be up to date. It’s very freeing.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [January 8, 2021, 6:02pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/9 "2021-01-08T18:02:55Z")

</div>

> [@misha\_mikhasenko](#):
>
> what is your workflow in Julia?

Bold of you to assume I only have one workflow, XD.

It depends.

1. If it is just a test/prototype that I can end up throwing in the trash by the end of the day I just create folder, active an environment, install the necessary packages and use a single jl file with everything inside `main`. If small enough I do not even start with git.
2. An intermediary case is data analysis for a paper. In this case the git repository is my PhD repository where I store all papers I am working on. I just create a subfolder in the latex folder of the paper and use Jupyter.
3. If it is something I will dedicate some time (weeks, months, …) and is not just data analysis, then I already start by using `PkgTemplates` to start a package (I do not remember if this already makes it a git repository, but I make sure it is a git repo). Does not matter if I intend to register it after or not, I do because being a package allows me to `dev` it in the general environment and `import` it in short scripts I may write.

If (1) grows it becomes (3) and if (3) grows I start dividing it in submodules. Basically, I use _CameCase_ for the `.jl` files that are a submodules (separation of namespace) and _snake\_case_ for the ones that are not submodules (i.e., that are just part of a module/namespace but separated for ease of search and reading). If a submodule is not a single file (i.e., it is broken in sub-submodules or has code separated in multiple _snake\_case.jl_ files) then they get their own folder (inside `src`), recursively. _snake\_case.jl_ are included a single time inside respective modules. _CamelCase.jl_ are often all included also a single time but at the start of the main package module, and if they have dependencies for each other they just `import` by means of the parent module (e.g., `MyPkg` has submodules `A` and `B`, that are `include`d just after `module MyPkg` line, and if `B` needs something from `A`, then inside `module B` I do `import ..A: name_of_function_needed`).

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 8, 2021, 6:19pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/10 "2021-01-08T18:19:55Z")

</div>

This is a constant struggle for me too. I’m starting to become more and more a fan of starting with creating a package straight away and create everything from the REPL. Then, tools like LaTeX or Pandoc can pick the output up. For me, scripts will always lead to lots of code duplication and the problem of naming the scripts. Simply put: the benefit of just having a package is that all the logic is split into functions instead of files, so it is more versatile. Added benefits are the included tools for

- testing
- package version control (via `[compat]` or `Manifest.toml`)
- documentation generation

Also, packages work well with the REPL and `Revise.jl`. With scripts, I’m always having to manually ensure that the state of the loaded code is the same as the state of the running code. Finally, working in package has standard solutions for most of your suggestions too because package developers also have to deal with writing test, scaling and passing parameters between different parts.

> [@misha\_mikhasenko](#):
>
> how do you save an intermediate representation?

For me, running time is not really a thing, so I would just ensure that I can reproduce it by going back in Git’s history.

But, as I said, I’m also struggling with this so there might be better approaches.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 9:13pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/11 "2021-01-08T21:13:10Z")

</div>

> [@pdeffebach](#):
>
> Here’s what I do.

That sounds interesting, I would like to understand it better. Do you have any public examples to look at?  
Do files that are `include`-d do some work or only define functions?

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 9:26pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/12 "2021-01-08T21:26:11Z")

</div>

> [@Henrique\_Becker](#):
>
> It depends.

It is all clear about (1) and (2).  
The case (3) is what I am interested in the most. My projects are of several months long.

What you do with modules and submodules sounds interesting. What is the use case for the submodules?  
I often struggle with placing functions of different generality in modules. For most of the functions, I have a general version and one, less general that takes some `const` object in. Maybe it is exactly the use case for the submodule … (?)

---

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [January 8, 2021, 9:28pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/13 "2021-01-08T21:28:27Z")

</div>

> [@misha\_mikhasenko](#):
>
> I am currently struggling to find a way to run the whole analysis that is a chain of different scripts with intermediate outputs. Any experience?

We use [dvc.org](http://dvc.org) in our tech stack. This is a tool built on top of git that versions your data, controls data provenance, ensures reproducibility, and maintains your computational pipelines.

Generally, our file directory has a `scripts/` folder which contains scripts that call functions from a module defined in `src/`. The outputs of those scripts gets stored in subfolders in a `data/` folder, and those outputs are tracked by `dvc` (as well as the commands used to run the script, and any inputs). Data is stored in any number of formats - could be `.csv` or `.sqlite` or a binary format that Julia can directly read like `BSON`.

In `dvc`, we mark the inputs and outputs to each particular stage, and `dvc` then builds up a DAG which charts the computational pipeline. If some intermediate stage has changed, `dvc` detects that change and re-runs anything that it depends on (as you specify).

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [January 8, 2021, 9:29pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/14 "2021-01-08T21:29:28Z")

</div>

I’ve summarised my own workflow [here](https://discourse.julialang.org/t/scripting-like-a-julian/50707) in the past. It basically comes down to minimizing performance problems & maximizing convenience/adaptability for me. Version management is still done by hand though, but all the fancy stuff your editor does for you anyway should work out of the box. I definitely use git repositories for every module though.

I don’t usually use submodules. If it’s important enough to factor out into a submodule, it’s usually important enough to make it its own thing.

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 9:30pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/15 "2021-01-08T21:30:54Z")

</div>

> [@rikh](#):
>
> I’m starting to become more and more a fan of starting with creating a package straight away and create everything from the REPL. Then, tools like LaTeX or Pandoc can pick the output up.

Interesting! How does it work precisely?

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 8, 2021, 9:39pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/16 "2021-01-08T21:39:15Z")

</div>

Could you tell me which parts are (the most) unclear? Then, I’ll elaborate on those

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 9:50pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/17 "2021-01-08T21:50:49Z")

</div>

Wow, sounds just fantastic! Maybe that is what I am looking for.  
Thanks! going to try on the weekend 🙂

---

<div class="post-metadata">

**Author:** ![misha\_mikhasenko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/misha_mikhasenko/32/5060_2.png) [@misha\_mikhasenko](https://discourse.julialang.org/u/misha_mikhasenko)\
**Post date:** [January 8, 2021, 9:54pm UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/18 "2021-01-08T21:54:17Z")

</div>

I was just confused about how LaTeX or Pandoc can pick up the output?  
What kind of output?  
Are you creating plots with PGFPlots? or maybe generating the text of your papers with Julia 🙂 ?

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 9, 2021, 8:34am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/19 "2021-01-09T08:34:12Z")

</div>

Dvc sounds cool. What do you think about it? I dislike it when tools work for the most part but don’t allow me to do fine-tuning (for example, when a tool manages citations but sometimes does it wrong). Does that happen with dvc?

Edit: And is it difficult to setup / maintain? I ask because I see mentions of databases, config files and cloud storage.  
Edit2: Do you also know how it differs from [Pachyderm](https://www.pachyderm.com/)?

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [January 9, 2021, 8:35am UTC](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030/20 "2021-01-09T08:35:23Z")

</div>

Ah, yes. Plots and tables. Latexify.jl has some nice conversions from DataFrame to other formats

[Next page](https://discourse.julialang.org/t/exploratory-research-project-workflow/53030.md?page=2)