# \[ANN\] DataToolkit.jl — Reproducible, flexible, and convenient data management
**URL:** https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757
**Category:** Package Announcements
**Tags:** package, announcement, data, reproducibility
**Created:** [October 9, 2023, 4:46pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757 "2023-10-09T16:46:00Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 9, 2023, 4:46pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/1 "2023-10-09T16:46:00Z")
[](https://github.com/tecosaur/DataToolkit.jl)
#### Reproducible, flexible, and convenient data management
* * *
[DataToolkit.jl](https://github.com/tecosaur/DataToolkit.jl) is part of a trio of packages that aim to provide an exceptionally convenient and extensible approach to data management. Compared to pipelines (e.g. nextflow, snakemake) this focuses more on helping when you want to make the data handling in an individual project/one-shot analysis easy and reproducible.
It subsumes the capabilities of (a non-exhaustive list):
- [DataDeps.jl](https://github.com/oxinabox/DataDeps.jl)
- [DataSets.jl](https://github.com/JuliaComputing/DataSets.jl)
- [RemoteFiles.jl](https://github.com/helgee/RemoteFiles.jl)
Here’s a taster of what using it looks like:
[](https://asciinema.org/a/DuKIkwJf78EmWLhl95kKRe5qg)
See the [Introduction in the docs](https://tecosaur.github.io/DataToolkitDocs/ref/stable/) and/or my JuliaCon23 presentation [[slides](https://pretalx.com/media/juliacon2023/submissions/9BTTRL/resources/presentation_0F8lVWc.pdf)] to learn more 🙂
[ ](https://www.youtube.com/watch?v=Ytz3JPgvOnE)
I’ve just released v0.7 🎉, and pending development/feedback I plan on tagging 1.0 late this year. I have a few things I want to do before then such as:
- Changing the checksum to KangarooTwelve/Blake3
- Settle on a public API
- More docs
- More tests
- More supported storage backends (artifacts, S3)
If this sounds of interest, please give it a whirl and let me know if you have any feedback, I’d be keen to hear thoughts/experiences with this 😀.
---
### Author:  [@adamslc](https://discourse.julialang.org/u/adamslc)
#### Post date: [October 9, 2023, 7:12pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/2 "2023-10-09T19:12:44Z")
This looks very interesting! Could you give a quick summery of the similarities/differences compared to [DrWatson.jl](https://github.com/JuliaDynamics/DrWatson.jl)?
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 10, 2023, 12:44am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/3 "2023-10-10T00:44:53Z")
To [quote George](https://github.com/JuliaDynamics/DrWatson.jl/issues/37):
> the mission with DrWatson is to keep things as simple as possible. And DrWatson is not so much a data management system, but a “scientific project assistant” let’s say.
So it’s a bit like being asked to compare DataFrames.jl and MLJ.jl — both “do things” with tables but have entirely different concerns.
That said, this could probably be used _in_ a DrWatson project, and maybe even integated more directly.
---
### Author:  [@jules](https://discourse.julialang.org/u/jules)
#### Post date: [October 10, 2023, 5:47am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/4 "2023-10-10T05:47:34Z")
How does this system deal with the versions of the loader packages? It seems to me like when CSV is needed, you’re prompted to install it in the usual way, giving you a manifest entry. But the data.toml doesn’t know about the version, so it would not be enough to be sure you get the same data on a different machine after deserialization. I guess checksumming can only apply to the raw bytes, so how can a user be sure that data is loaded as originally intended? Or is the system intended to work together with your manifest, so that only the combination of data.toml plus manifest.toml suffices to describe what you did.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 10, 2023, 6:07am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/5 "2023-10-10T06:07:49Z")
Exactly what you suppose at the end. `Data.toml` is expected to live _within_ a Julia project (next to the `Project.toml` and `Manifest.toml`). We let `Pkg.jl` do the excellent job it already does at reproducing package environments, and together with `DataToolkit` you get complete reproducibility of a project that involves loading data 🙂
Hmm, maybe it would be good to have an example repo showing an example data project and data package built using `DataToolkit`… 🤔. For the first of these, I could just save the end result of [the tutorial](https://tecosaur.github.io/DataToolkitDocs/ref/stable/tutorial/).
---
### Author:  [@jules](https://discourse.julialang.org/u/jules)
#### Post date: [October 10, 2023, 7:23am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/6 "2023-10-10T07:23:50Z")
Ok so it’s less of a standalone data description than I first thought. But I guess that makes sense, given the many options data loading packages expose and which can be subject to change.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 10, 2023, 8:01am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/7 "2023-10-10T08:01:05Z")
The only truly “standalone” way of handling data is a tool that never changes and just does basic downloads and maybe a checksum on a pile of bytes.
If you want to go beyond this (as `DataToolkit` tries to), you need to grapple with the fact that the data becomes intertwined with the way it is loaded/processed. Some tools (e.g. DataLad) let you make this reproducible by containerising the processing step(s), but if we indulge ourselves and make a Julia-specific tool we can just use `Pkg.jl` 😃.
It’s also worth noting that the version of `DataToolkit` (+ dependencies) will itself also be managed by `Pkg.jl` of course, and in this way everything is (IMO) rather nicely bundled up.
In future, it could be nice to have a way to concatenate `{Project,Manifest,Data}.toml` into a single file that _would_ be as portable as you seem to be thinking of. That said, `.zip` and `.tar` files already exist…
All that said, barring drastic changes to the loaders used (e.g. `CSV.jl`) the data files should be rather portable, and if you want to go a step further you can create a package that has a `Data.toml` along with a `Project.toml`. That’s a pretty good way of sharing datasets (since you can then just do `using SomeDataSets` and then … use them).
---
### Author:  [@mthelm85](https://discourse.julialang.org/u/mthelm85)
#### Post date: [October 10, 2023, 3:31pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/8 "2023-10-10T15:31:21Z")
I have a few datasets that I use across multiple projects and, depending on the project, I only ever need to load a subset of all the available columns. When loading a dataset with DataToolkit.jl, is there a way to tell it which columns from a tabular dataset that I want to load? Something like the `select` keyword argument from CSV.jl?
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 10, 2023, 3:45pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/9 "2023-10-10T15:45:31Z")
Not OOTB, but there are two ways you could do so:
- Via derived data sets (e.g. create `a-1` which is column `1` of `a`)
- Via a plugin that adds that behaviour (plugins can do a _lot_)
---
### Author:  [@DanielVandH](https://discourse.julialang.org/u/DanielVandH)
#### Post date: [October 11, 2023, 4:20am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/10 "2023-10-11T04:20:30Z")
Great work! I really like the data REPL - very creative. The JuliaCon talk is great too, easy to follow - the part at 10:00 is awesome.
---
### Author:  [@NicholasWMRitchie](https://discourse.julialang.org/u/NicholasWMRitchie)
#### Post date: [October 11, 2023, 4:58pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/11 "2023-10-11T16:58:12Z")
DrWatson.jl is more about creating well-organized, reproducible environments for performing data analysis. It builds off the package manager and uses a custom environment to ensure all calculations can be replicated in a consistent package environment. DataDeps.jl can be used to import the data into a DrWatson project. This can be very convenient when the data is bulky and/or must be stored according to a journal’s or your organization’s requirements. As a government scientist, I publish the data my organization repository (made findable through [data.gov](http://data.gov)) and then a DrWatson project on Github to process the data.
They are both super useful.
---
### Author:  [@Datseris](https://discourse.julialang.org/u/Datseris)
#### Post date: [October 12, 2023, 9:17am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/12 "2023-10-12T09:17:12Z")
I agree. This sounds very promising. I’ve missed this announcement post but I’ll try to read in more detail and give more feedback. I am also happy to give support to better/more/simpler integration between the two packages!
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [April 7, 2024, 10:18am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/13 "2024-04-07T10:18:09Z")
With **v0.9 just released** , DataToolkit continues to support more file formats, with fewer bugs 😛.
If you haven’t tried it out, or did before but quickly ran into an issue, I’d encourage you to give it another look. A few people have raised bugs/usability issues on GitHub, and that’s helped me improve the state of the project 🙂.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [June 16, 2024, 8:21am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/14 "2024-06-16T08:21:16Z")
A lot of work is going into the next release ( **v0.10** ), to give a sneak peek (and since there’s no changelog 😅), here’s what I’ve done so far:
- Bumping the minimum Julia to 1.9 and embracing package extensions
- Replacing `@import` with `@require`: [d9e9226](https://github.com/tecosaur/DataToolkit.jl/commit/d9e92265a95bb16452371070e5486e565c7cbdd5)
- Removal of the `SmallDict` type (the original issue is a lot better with `Memory` in 1.11): [98a6723](https://github.com/tecosaur/DataToolkit.jl/commit/98a6723e68de81689ffb972bb3c1cfab8da04ecb)
- Improved type inference/logic: [389df28](https://github.com/tecosaur/DataToolkit.jl/commit/389df28b3a6922c6ff9b45a3165713cec91c0918)
- Separate the REPL mode and Store out into new packages
- Rename `DataToolkitBase` to `DataToolkitCore`
- Split `DataToolkit` into a more user-facing `DataToolkit` and package-facing (new) `DataToolkitBase`
- Move all these packages into [DataToolkit.jl](https://github.com/tecosaur/DataToolkit.jl) monorepo
- Improved load time and precompilation
- Support for opening files as a `FilePathsBase.AbstractPath`
- Support for (basic) S3 downloads
- More image types (gif, webp)
- Logging capability moved from `DataToolkitCommon` to `DataToolkitCore`. It works a bit differently (IMO, better) and is now configured by `Preferences`
- Support working with directories as well as files, without compromising data integrity (thanks to cached merkle tree checksumming): [e413116](https://github.com/tecosaur/DataToolkit.jl/commit/e4131165cf1fcd5eb1a6fecdbdae15939ce4a28f)
- WIP documentation improvements
Of these, I’d say the “headline” changes would be:
- Directory support
- Package restructuring to make it better suited for package-usage by having a new `DataToolkitBase` that doesn’t provide the `data>` REPL
- The move to a monorepo
There’s still a good bit of work needed before I’m confident enough to cut the v0.10 release. In particular, I’m worried about new bugs with all the code changes. Once I’ve tinkered and tested this a bit more, I’ll see about doing so.
In the meantime, this is a great time to request design/feature/API tweaks and report bugs! Don’t hesitate to shoot me a message here, on slack, or zulip 🙂.
---
### Author:  [@HanD](https://discourse.julialang.org/u/HanD)
#### Post date: [June 20, 2024, 9:50am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/15 "2024-06-20T09:50:30Z")
Hi, @tecosaur!
`DataToolkit` looks great, I’ll give it a spin some time in the future.
* * *
The monorepo layout you went for really piques my interest. I’m thinking of starting a similar monorepo in my work, but I have serious doubts how well Julia supports this layout. Perhaps you can shed some light on how you tacked the following issues:
- There is a so-called [package directories environment](https://docs.julialang.org/en/v1/manual/code-loading/#Package-directories), the support doesn’t go far enough. In particular, `Pkg` doesn’t seem to be aware of this environment at all. In particular, I have problems with:
- One can use `Pkg.dev` to refer to packages within the monorepo, but that information is stored in the `Manifest.toml` file, which thus needs to be committed, and it’s also intransitive (i.e., a third package `C` using `B` will also have trouble finding `A`, unless the latter is added explicitly with `dev`).
I see in your monorepo source, that you refer to sibling packages just like any other package, so I assume you plan on registering them all in the `General` registry. Which is fine, I guess, but then how you work on WIP changes that affect multiple packages? They can’t be registered until the changes are merged, but they don’t compile (and thus cannot be tested and merged) until the new versions aren’t registered in the registry. Seems to be a Catch-22 to me. Do you use lots of `Pkg.dev`s while in development? Or do you have an internal test registry to which you register versions still in development?
Any insight is welcome!
---
### Author:  [@jules](https://discourse.julialang.org/u/jules)
#### Post date: [June 20, 2024, 3:38pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/16 "2024-06-20T15:38:37Z")
> [@HanD](#):
>
> Which is fine, I guess, but then how you work on WIP changes that affect multiple packages? They can’t be registered until the changes are merged, but they don’t compile (and thus cannot be tested and merged) until the new versions aren’t registered in the registry.
The monorepo makes this easier actually, because in your CI run you first check out the repo, that means the whole repo, so all sibling packages with their changes from that branch. You just have to dev them first because just running `]test` would not pick them up from the local folder.
>
In future Julia versions, I think one will be able to specify sources for packages in a Project.toml which should enable a simpler workflow where you specify directly that your test env depends on the local packages around it.
---
### Author:  [@merlin](https://discourse.julialang.org/u/merlin)
#### Post date: [July 6, 2024, 7:50pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/17 "2024-07-06T19:50:13Z")
> [@tecosaur](#):
>
> parti
Hi! I just learned about this and I’m trying it out. I have rolled my own package to organize the bazillion datasets I work with, and I’m ready for something much smarter – this looks great!
Questions coming to me after reading the docs:
**What happens when I’ve added thousands of datasets?**
What I mean is, any project might need to work across several arbitrary datasets. Therefore with a new ticket I would start a new project folder and would like to quickly reference any one of my datasets without doing this every time:
```jl
data> add PBB-198.customer_submits_msa_checkbox s3://muh_data/csv/dataset=pbb_198.customer_submits_msa_checkbox/rundate=2024-07-23/platform=mobile/data.csv
```
So I guess if I’m working in subdirectories of a single Data.toml project, that file is going to grow into MB size. Will that be an issue?
**Can it handle partitions?**
You mentioned ‘directory support’, does that mean datasets of multiple files across directories that represent partitions?
```julia
s3://muh_data/csv/dataset=customer_orders/rundate=2024-07-23/platform=mobile/data.csv
s3://muh_data/csv/dataset=customer_orders/rundate=2024-07-23/platform=web/data.csv
```
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [August 20, 2024, 9:57am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/18 "2024-08-20T09:57:19Z")
Seems like I missed the new reply notification somehow @merlin, but let’s see if I can help answer some of your questions 🙂
> [@merlin](#):
>
> What happens when I’ve added thousands of datasets?
> [@merlin](#):
>
> So I guess if I’m working in subdirectories of a single Data.toml project, that file is going to grow into MB size. Will that be an issue?
I’m not sure! For what I think you’re describing, it does sound like “one big Data.toml” might be what you want, but scaling isn’t something I’ve looked at much (I’ve had at most maybe a hundred datasets in a project at once). That said, I hope DataToolkit is fine with a few thousand datasets, and if it isn’t that’s something I’ll want to fix!
> [@merlin](#):
>
> **Can it handle partitions?**
> You mentioned ‘directory support’, does that mean datasets of multiple files across directories that represent partitions?
Not currently OOTB, I’m guessing what you mean by “partitions” is one folder of content that’s had it’s files split across multiple partial copies of the folder?
If this is correct, then I think I can see a way to implement merging of multiple “partial” folders using symlinks that should be fairly straightforward.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [October 4, 2024, 2:46am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/19 "2024-10-04T02:46:44Z")
> [@tecosaur](#):
>
> I’ve had at most maybe a hundred datasets in a project at once
Update, I’m now working with a project that has ~20k datasets. It takes a moment to load, but seems completely fine.
---
### Author:  [@merlin](https://discourse.julialang.org/u/merlin)
#### Post date: [October 4, 2024, 3:31pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/20 "2024-10-04T15:31:38Z")
This is awesome info. It may seem like an edge case up front, but its one of those hard to unwind decisions that gave me pause before adopting this idea. Its so well thought out – thank you!
[Next page](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757.md?page=2)