# DrWatson - the perfect sidekick to your scientific inquiries!

**URL:** <https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241>\
**Category:** Package Announcements\
**Created:** [April 17, 2019, 2:51pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241 "2019-04-17T14:51:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 17, 2019, 2:51pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/1 "2019-04-17T14:51:02Z")

</div>

![DrWatson-logo](https://global.discourse-cdn.com/julialang/original/3X/1/7/17f50b979a80661005a92c8e49a0bd06433a208d.png)

Dear All,

I am very happy to announce the first beta version of our new package, called DrWatson:

> **[Introduction · DrWatson.jl](https://juliadynamics.github.io/DrWatson.jl/dev/)**
>
> Documentation for DrWatson.jl.

## Motivation

DrWatson is the result of some scientists being “fed up” with the hardship of handling scientific projects. We want to stop repeating the same processes for every project, stop screaming our lungs out in frustration, and make handling our scientific projects easier. Things like:

- Urgh, I moved my folders and now my load commands don’t work anymore!
- Do I have to produce a dataframe of my finished simulations AGAIN?!
- PFfffff I am tired of typing `savename = "w=$w_f=$f_x=$x.jld2`, can’t I do it automatically?

are what we want to battle.

## Why this can really help you

I believe that DrWatson is a package that can truly make your life easier as a scientist, by removing the annoying parts of managing a project.  
In the [Description](https://juliadynamics.github.io/DrWatson.jl/dev/#Description-of-DrWatson-1) page of the documentation of DrWatson you find the core principles of the package:

1. **Non-Invasive.**
2. **Simple.**
3. **Consistent.**
4. **Allows increments.**
5. **Reproducible.**
6. **Modular.**
7. **Scientific.**

Want I want to really focus on are the first two points. Many other similar approaches exist, that aim to support scientific project management (see the “Inspirations” section of our documentation) but what I have come to notice is that they suffer from a common problem: they just aren’t simple enough. You have to do too much work to use them!

DrWatson takes a radically new approach: instead of complicated pipelines that you have to follow to benefit, DrWatson only asks you to just use a couple of functions.

### Example 1: [`savename`](https://juliadynamics.github.io/DrWatson.jl/dev/name/#Naming-Simulations-1)

As a first example, lets look the very common situation of using variable values to create file names. E.g. you have the case of running a simulation with variables `a=3, b=5, model="water"`. Typically, when saving the simulation data resulting from your script you would consider writing a name `prefix_a=$a_b=$b_model=$model`. At some point you might be frustrated with having to do this all the time, and you might write down a function that takes in a dictionary and produces such a string.

This is what the function `savename(c)` from DrWatson does. It transforms any key-based Julia container `c` (Dict, NamedTuple, Composite Type) into a string like the one above. And besides of doing what you want, it also is deterministic and allows for customization.

(of course, you don’t have to use the customization aspects, this is where the **Modular** aspect of DrWatson shines)

### Example 2: [`tagsave`](https://juliadynamics.github.io/DrWatson.jl/dev/save/#Tagging-a-run-using-Git-1)

Wouldn’t it be awesome if **every saved datafile** contains a record of the Git commit of the project, when the file was saved? Wouldn’t it be awesome if achieving this required **no additional effort**?

If in your code you replace the function `save(file, data)` with `tagsave(file, data)` then the saved file will have one additional field called `:commit`, which will contain the commit ID of your project when you saved the file. And without writing a single extra line of code, all of your saved data tell you the commit they come from! 🙂

Of course, this requires your scientific project to be a Git repository as well. Well, no worries, since DrWatson also offers a simple template for a scientific project which is also a Git repository.

## Beta Invitation

**We actively looking for beta testers and contributors!**

Please, consider using DrWatson. Consider helping us in the development and please report problems, and give us ideas and feedback by using the [GitHub page](https://github.com/JuliaDynamics/DrWatson.jl) of the repository.

DrWatson is part of JuliaDynamics, so you can also chat in the channel `#dynamics-bridged` of the Julia Slack.

* * *

_logo developed by @cormullion (of course)_

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [April 17, 2019, 3:20pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/2 "2019-04-17T15:20:56Z")

</div>

This is neat - I’ve been contemplating something parallel to this idea. The `tagsave` concept is definitely critical, but it seems like this is for saving data blobs? What if I want to generate a CSV file or a figure image from a plot? My idea was to have an automatic generation of a `data.toml` file or something that logs dates, commits and other metadata linked to files, could this do something similar?

My other major requirement is the generation of docs-like files (pdf/html) from notebooks, either from Weave or Literate files, and re-running or doing some sort of testing to make sure code in old notebooks is kept up to date. Any thoughts on that end?

In any case, I’d definitely be interested in testing/contributing.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [April 17, 2019, 3:27pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/3 "2019-04-17T15:27:59Z")

</div>

Is this something similar to [https://ropensci.github.io/drake/](https://ropensci.github.io/drake/) ?

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 17, 2019, 3:56pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/4 "2019-04-17T15:56:24Z")

</div>

@kevbonham

> [@kevbonham](#):
>
> The `tagsave` concept is definitely critical, but it seems like this is for saving data blobs? What if I want to generate a CSV file or a figure image from a plot?

Adding the commit ID can be done as long as the file you save is a Dictionary. `tagsave` is just a lightweight wrapper around `FileIO.save` but it still needs to somehow add the commit id. I do not know of another way to do it generally (e.g. for an image; how to “add a string” to an image?).

> [@kevbonham](#):
>
> My idea was to have an automatic generation of a `data.toml` file or something that logs dates, commits and other metadata linked to files, could this do something similar?

Seems like a very valid suggestion, please go ahead and open an issue to discuss this further!

> [@kevbonham](#):
>
> My other major requirement is the generation of docs-like files (pdf/html) from notebooks, either from Weave or Literate files, and re-running or doing some sort of testing to make sure code in old notebooks is kept up to date. Any thoughts on that end?

I don’t really use these approaches for scientific purposes so I don’t have any experience… But this is exactly why we are looking for beta testers and users and people to contribute opinions.

@mkborregaard

> [@mkborregaard](#):
>
> Is this something similar to [https://ropensci.github.io/drake/](https://ropensci.github.io/drake/) ?

No, from the 6-minute video I just watched they don’t seem similar. DrWatson also seems to have a much simpler and cleaner approach to helping you (e.g. you don’t have to make a “DataFrame” out of every function in your code!)

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [April 17, 2019, 4:26pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/5 "2019-04-17T16:26:25Z")

</div>

Cool to see that you are making such rapid progress on this!

Nice would also be an integration with the input side of modeling. Maybe some integration with DataDeps.jl?

---

<div class="post-metadata">

**Author:** ![npr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/npr/32/10626_2.png) [@npr](https://discourse.julialang.org/u/npr)\
**Post date:** [April 17, 2019, 4:29pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/6 "2019-04-17T16:29:29Z")

</div>

> [@kevbonham](#):
>
> My idea was to have an automatic generation of a `data.toml` file or something that logs dates, commits and other metadata linked to files, could this do something similar?

For saving metadata along with the contents, it might be worth adopting [JLSO](https://github.com/invenia/JLSO.jl). It wraps other formats (currently [BSON](https://github.com/MikeInnes/BSON.jl)), while storing key metadata such as the version of julia and all involved packages that were used in creating the file.

---

<div class="post-metadata">

**Author:** ![jessebett](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jessebett/32/2006_2.png) [@jessebett](https://discourse.julialang.org/u/jessebett)\
**Post date:** [April 17, 2019, 8:10pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/7 "2019-04-17T20:10:21Z")

</div>

Default project setup could use a `test` directory to encourage people to write unit tests on their `src`.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 17, 2019, 8:29pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/8 "2019-04-17T20:29:10Z")

</div>

Thanks, please open suggestions on the GitHub page so that it is easier to keep track for developers! 🙂

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [April 17, 2019, 8:45pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/9 "2019-04-17T20:45:20Z")

</div>

Interesting. I have found that “stashing” data is a surprisingly difficult problem which I still don’t have a solution for. This seems like it should be very simple but in practice I’ve always found it hard, in part because it’s hard to decide on exactly what the “metadata” is, but also because it’s hard to guarantee that said metadata gets propagated everywhere that something needs to be saved (which is perhaps a sign that I should give up and just be willing to use some global variables). It’s also interesting that this is one of the few problems that seems very much the same in my current job as a data scientist as it did during my career in physics.

I’m eager to take a look at this and see to what extent it suits my needs.

By the way, a “new variant” of this problem for me is that now I frequently have to save to S3 buckets rather than the filesystem (whether actual or emulated). Do you allow saving and loading to arbitrary IO buffers with arbitrary serialization methods?

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [April 17, 2019, 9:34pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/10 "2019-04-17T21:34:36Z")

</div>

> [@Datseris](#):
>
> (e.g. for an image; how to “add a string” to an image?).

Right, I don’t think this would work, hence the ideas about having a toml or something that keeps track.

Another ignore that occurs to me - you have to be sure you tagsave _after_ committing the code that makes it. This issue usually the reverse of my workflow. I typically write code, try a free things, then once I get a result, commit. Presumably this would mean my tagsaved file would have the hash of the previous commit.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 17, 2019, 9:45pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/11 "2019-04-17T21:45:43Z")

</div>

@ExpandingMan

Metadata is an area we have not developed DrWatson a lot, mostly because we are considering adding an API to CaosDB, a new Database management system. DrWatson is more about managing your scientific project and less about data management, at least at this present state. The data management is as basic as it can get (but still useful I feel).

> [@ExpandingMan](#):
>
> Do you allow saving and loading to arbitrary IO buffers with arbitrary serialization methods?

There are no restrictions by DrWatson: you continue to use your project as you see fit. Now, if you want to use the function `tagsave`, well, it is a wrapper around `FileIO.save`. This was the only way we could make it general enough. I do not know the full extend of the capabilities of FileIO, but it should not be hard for us to dispatch on different IO buffers before calling `save`. Open an issue if you have a concrete idea/suggestion!

@kevbonham

> [@kevbonham](#):
>
> This issue usually the reverse of my workflow.

Yeah unfortunately this is most often the case for most of us… 😃 But it is bad practice. And writing a package that takes care of this for you automatically would be too invasive. For now, if you try to get the commit ID of a dirty repo, DrWatson throws a warning (not error!) and also adds the string `_dirty` to the commit ID, see the docs around `tagsave`.

This somewhat compensates for the problem you mention.

---

<div class="post-metadata">

**Author:** ![grero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grero/32/109_2.png) [@grero](https://discourse.julialang.org/u/grero)\
**Post date:** [April 18, 2019, 1:43am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/12 "2019-04-18T01:43:50Z")

</div>

This is really and something I’ve been thinking about a lot has well. I was using [Sumatra](https://pythonhosted.org/Sumatra/) for a while, which worked pretty well, but since it’s written in Python, it required a bit of a context switch every time I used it. The requirement to commit my code before saving data also became kind of a pain, even though as was pointed out by others, it is good practice. I like the idea of throwing a warning and adding the \_dirty tag.  
I will definitely give this a try. Thanks!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [April 18, 2019, 8:14am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/13 "2019-04-18T08:14:03Z")

</div>

> [@Datseris](#):
>
> how to “add a string” to an image?

Just pinging in, there is a standard way to add almost any information to images: in their metadata. A related effort was discussed in [this post](https://discourse.julialang.org/t/programmatic-image-tagging/6581). So for an actual image, yea you can add a string (among other things). But this is a specific solution for images, maybe a more general solution (like the TOML idea of @kevbonham) is better.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 18, 2019, 9:09am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/14 "2019-04-18T09:09:44Z")

</div>

Arggghh. Meaningful file names, and meaningful directory names. Arrrgggh.  
I suppose there’s no stopping them, but we are in an era of S3 compliant object stores.  
One day we will get away from huge deep directory trees where the directory names are the metadata.

Look at Architecta, for instance (commercial product)

> **[Arcitecta - Big Data & Data Management Software & Solutions](https://www.arcitecta.com/)**
>
> Arcitecta's big data & data management software solutions lets you leverage the power of metadata to exploit massive data sets to unlock rich insights & speed

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 18, 2019, 9:10am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/15 "2019-04-18T09:10:05Z")

</div>

The Tagsave thing I really like though!

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2019, 9:21am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/16 "2019-04-18T09:21:43Z")

</div>

> [@johnh](#):
>
> One day we will get away from huge deep directory trees where the directory names are the metadata.

I don’t agree with this approach. I’d much rather have my data in the folder of my science project, in a clear and well organized folder tree. I know what each folder means. The arcitecta that you shared is already too complicated for me. Only the fact that it uses a server to handle data is already too complicated for me.

I think there are two things to separate here:

1. A specific, contained scientific project (which of course could use data from other projects)
2. A huge collaboration with many projects, many many people and absurd amount of data.

For example, I do the first, and DrWatson has also the clear goal of helping the first. DrWatson it is not a data management system (but as I said, we plan to interface CaosDB, which _is_ a data management system tailored for scientific projects).

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2019, 9:30am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/17 "2019-04-18T09:30:19Z")

</div>

I’ve added a note in the docs clarifying the data management part (which is not the purpose of DrWatson) and mentioning CaosDB (whos purpose is data management).

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [April 18, 2019, 10:04am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/18 "2019-04-18T10:04:40Z")

</div>

But doesn’t CaosDB fall into the too-complicated category as well? Here a fig from their paper:

 ![caosdb](https://global.discourse-cdn.com/julialang/original/3X/3/b/3ba4262ba55d52da5fbccbaa67892bec7d20d35a.png)

I’ve been procrastinating reading up on [https://www.datalad.org/](https://www.datalad.org/), which was posted by @c42f in [Data versioning · Issue #37 · JuliaDynamics/DrWatson.jl · GitHub](https://github.com/JuliaDynamics/DrWatson.jl/issues/37). Based around git, this might be a better fit for Julians?

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2019, 10:14am UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/19 "2019-04-18T10:14:27Z")

</div>

Yes, of course, it absolutely is too complicated. Which is why it will a separate package if we manage to port it to Julia.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 18, 2019, 12:57pm UTC](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241/20 "2019-04-18T12:57:11Z")

</div>

@Datseris Thankyou for separating out the problem like that. I agree.

[Next page](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241.md?page=2)