# State of the Art for Data Version Control?
**URL:** https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246
**Category:** General Usage
**Tags:** question, data
**Created:** [November 14, 2022, 4:17pm UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246 "2022-11-14T16:17:30Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)
#### Post date: [November 14, 2022, 4:17pm UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/1 "2022-11-14T16:17:30Z")
Hi folks!
I am working with DrWatson.jl and am wondering: is there any suggested tools or workflows to handle data version control within Julia? I saw that there was [DataSets.jl](https://docs.juliahub.com/DataSets/4adr3/0.2.7/) made by @c42f and some ongoing [discussions](https://github.com/JuliaDynamics/DrWatson.jl/issues/255) that @Datseris was having. Wasn’t sure what the current state of the art is. Thanks!
~ tcp 🌳
---
### Author:  [@Datseris](https://discourse.julialang.org/u/Datseris)
#### Post date: [November 15, 2022, 9:14am UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/2 "2022-11-15T09:14:01Z")
Hm, unfortunately there aren’t many options in JuliaLand. I have seen the DataSets.jl package as well, but personally I am not a fan: it’s too complicated for my liking.
In DrWatson there is an attempt to bring a metadata-attaching functionality, in a sense of attaching a declarative text file “toml” style to any dataset. This would give you full governance and in a sense version control since the metadata themselves could easily be version controlled. The ongoing attempt is described in [this issue](https://github.com/JuliaDynamics/DrWatson.jl/issues/316) and this [pull request](https://github.com/JuliaDynamics/DrWatson.jl/pull/333). Its called “DrWatsonSim”.
The third option I am aware of is [CaosDB](https://github.com/JuliaDynamics/CaosDB.jl): it is a scientific database management that has been test driven in several large scale collaborative scientific projects and would allow data management and provenance. However, it only has a barebones Julia implementation that hasn’t been updated in 3 years ☹ I haven’t used CaosDB in my work because, the way it is now, actually putting data on the CaosDB server would take too much effort. However, I am sure that there could be many very cool things happening like automatic integration with DrWatson workflows to put your data automatically in a database entry by using the configuration container given in `produce_or_load`. But at the moment the developers do not see Julia as worthy to allocate their resources on. I hope they change their mind at some poin, as Python doesn’t have something as cool as DrWatson…
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [November 15, 2022, 1:16pm UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/3 "2022-11-15T13:16:24Z")
Do you mean for things larger than the github file size limit or what? For things under the limit, I simply use github.
---
### Author:  [@svilupp](https://discourse.julialang.org/u/svilupp)
#### Post date: [November 16, 2022, 6:41am UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/4 "2022-11-16T06:41:08Z")
Have you considered non-Julia solutions?
If the data is public and not too large, I’d go with git.
For anything else, I’d suggest [DVC](https://dvc.org/doc/user-guide/overview#) - it seems quite popular.
You can think of it as git for data. It works well together with git. But instead of git tracking the data itself, git tracks only the dvc metadata/references and dvc itself gets you the actual files based on the metadata/references in your working tree.
---
### Author:  [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)
#### Post date: [November 29, 2022, 8:40pm UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/5 "2022-11-29T20:40:44Z")
Hey @Datseris 👋 thanks for your thoughts!
> [@Datseris](#):
>
> Hm, unfortunately there aren’t many options in JuliaLand. I have seen the DataSets.jl package as well, but personally I am not a fan: it’s too complicated for my liking.
Yea, I got a sense of it being complicated as well but I haven’t tried it fully yet. Perhaps I am just not reading how to use it correctly.
> [@svilupp](#):
>
> For anything else, I’d suggest [DVC](https://dvc.org/doc/user-guide/overview#) - it seems quite popular.
>
> You can think of it as git for data. It works well together with git. But instead of git tracking the data itself, git tracks only the dvc metadata/references and dvc itself gets you the actual files based on the metadata/references in your working tree.
I owe you a great thank you @svilupp ! DVC is the ticket for me and is perfect for my use case. Thanks! Having worked with it some, have to admit, it is a little cumbersome using the tool – will have to keep practicing with it!
---
### Author:  [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [November 30, 2022, 9:52pm UTC](https://discourse.julialang.org/t/state-of-the-art-for-data-version-control/90246/6 "2022-11-30T21:52:41Z")
DVC is great if you’re comfortable with git, but [Weights & Biases](https://wandb.ai) is a much more user friendly alternative IMO.