# Data storage/loading for data produced by algorithms and metadata

**URL:** <https://discourse.julialang.org/t/data-storage-loading-for-data-produced-by-algorithms-and-metadata/27029>\
**Category:** Data\
**Created:** [July 31, 2019, 12:19pm UTC](https://discourse.julialang.org/t/data-storage-loading-for-data-produced-by-algorithms-and-metadata/27029 "2019-07-31T12:19:43Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![platawiec](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/platawiec/32/31914_2.png) [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Post date:** [July 31, 2019, 12:54pm UTC](https://discourse.julialang.org/t/data-storage-loading-for-data-produced-by-algorithms-and-metadata/27029/2 "2019-07-31T12:54:05Z")

</div>

For this task, I use [DVC (Data Version Control)](https://dvc.org/). This is a language-agnostic tooling for managing data dependencies and pipelines. It requires a bit of discipline, but the idea is that “metadata” is really a general concept that goes beyond what you may typically consider (e.g parameters in a JSON or txt file) but also the scripts and environment that make that final data (which you can track with DVC by including the files and `Manifest.toml` as dependencies). You can then set up a pipeline on the output data by including it as a dependency in a subsequent stage. Changes to your metadata and resulting outputs are tracked in your git history.

Reproducibility and the ability to integrate with cloud storage is a huge thing for me, so I accept the overhead that using something like DVC induces. Because you’re working with smaller files and may need to be more nimble than me, I would recommend looking at [DrWatson.jl](https://github.com/JuliaDynamics/DrWatson.jl). Although it is not a data management system like DVC or others, for some use cases it can certainly help automate the tracking and saving of simulations, which may be all you need.

Also, see the [DrWatson.jl announcement](https://discourse.julialang.org/t/drwatson-the-perfect-sidekick-to-your-scientific-inquiries/23241) for further discussion of these types of toolings.

---

_[View the full topic](https://discourse.julialang.org/t/data-storage-loading-for-data-produced-by-algorithms-and-metadata/27029)._