# \[ANN\] DataToolkit.jl — Reproducible, flexible, and convenient data management
**URL:** https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757
**Category:** Package Announcements
**Tags:** package, announcement, data, reproducibility
**Created:** [October 9, 2023, 4:46pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757 "2023-10-09T16:46:00Z")
**Posts on this page:** 9
**Page:** 2
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 20, 2026, 8:32am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/21 "2026-07-20T08:32:18Z")
I’m going to put on a burst of effort to get v0.10 out the door, leaning on LLMs to accelerate the testing/bugfixing/documenting process.
I faintly recall some ideas for further features/enhancements, but at the moment the changelog is pretty much what I shared earlier. Restated for convenience:
* * *
## DataToolkit v0.10
### Highlights
- **Directory support** : datasets can now be whole directories, not just single files, with data integrity preserved via cached Merkle-tree checksumming.
- **Package restructuring** : `DataToolkitBase` was renamed to `DataToolkitCore`, and a new lighter-weight `DataToolkitBase` was introduced for _package authors_ (data dependencies without the `data>` REPL). The user-facing `DataToolkit` retains the full REPL experience.
- **Monorepo** : all packages (`DataToolkit`, `DataToolkitBase`, `DataToolkitCore`, `DataToolkitStore`, `DataToolkitREPL`, `DataToolkitCommon`) now live in a single `DataToolkit.jl` repository.
### New features
- Open files as a `FilePathsBase.AbstractPath`.
- Basic S3 download support.
- More image formats: GIF and WebP.
- REPL mode and the content store are now split into their own packages (`DataToolkitREPL`, `DataToolkitStore`).
### Improvements
- **Faster load times and precompilation.**
- **Better logging** : moved from `DataToolkitCommon` into `DataToolkitCore`, reworked, and now configured via Preferences.
- Improved type inference and dispatch logic.
- A _lot_ of bugfixes
- Documentation improvements (ongoing).
### Under the hood
- Minimum Julia version raised to **1.9** , adopting package extensions.
- `@import` replaced with `@require` for lazy package loading.
- Removed the internal `SmallDict` type (Julia 1.11’s `Memory` addresses the original motivation).
* * *
If there are any enhcancements you’d like to see, now is an excellent time to mention them 🙂
---
### Author:  [@langestefan](https://discourse.julialang.org/u/langestefan)
#### Post date: [July 20, 2026, 9:33am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/22 "2026-07-20T09:33:10Z")
Congrats! ps. some of the links in the original post are dead. Including the showcase video 🙂
---
### Author:  [@klafyvel](https://discourse.julialang.org/u/klafyvel)
#### Post date: [July 20, 2026, 2:00pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/23 "2026-07-20T14:00:29Z")
Thanks for the work on DataToolkit! If now’s the time to share a wishlist, I’d be amazing to be able to pull dependencies from a Google Doc folder, maybe prompting the user for some authentication when needed. Maybe this can be achieved via rclone integration?
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 20, 2026, 2:02pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/24 "2026-07-20T14:02:36Z")
It is a good time to share a wishlist!
For pulling from google drive, the two questions that would help are:
- Is there a good way to fetch from google drive in Julia?
- What’s a good way to stash credentials? Some sort of cross-platform keyring package seems like what we ideally want.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 20, 2026, 2:32pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/25 "2026-07-20T14:32:32Z")
Unfortunately [GitHub - alecloudenback/Keyring.jl · GitHub](https://github.com/alecloudenback/Keyring.jl) doesn’t look particularly lively
---
### Author:  [@klafyvel](https://discourse.julialang.org/u/klafyvel)
#### Post date: [July 21, 2026, 9:09am UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/26 "2026-07-21T09:09:14Z")
I know that GoogleDrive.jl exists, but I also remember some pain points in using it because I have never been able to make it download a complete folder, and the URLs are hard to guess, in opposition to rclone where you can have it download a folder based on its path within your own drive.
It also seems that someone made an interface to rclone in Julia, but I haven’t tried it: [GitHub - imohag9/RcloneInterface.jl: Julia interface to Rclone a command-line program to manage files on cloud storage · GitHub](https://github.com/imohag9/RcloneInterface.jl).
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 21, 2026, 4:18pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/27 "2026-07-21T16:18:42Z")
There are multiple services using authentication I would like to support, I just want to handle them well, and not in a “dump a secret into a plain file” type way.
Rclone seems like a good thing to look at interfacing with, and worst case I might end up having a stab at finishing Keyring.jl (for some future version).
---
### Author:  [@klafyvel](https://discourse.julialang.org/u/klafyvel)
#### Post date: [July 21, 2026, 4:45pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/28 "2026-07-21T16:45:06Z")
Oh yes, of course. Unfortunately I have zero knowledge on that part…
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 21, 2026, 4:46pm UTC](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757/29 "2026-07-21T16:46:27Z")
I can’t say I have much either, but I’ve never let that stop me before 😂
[Previous page](https://discourse.julialang.org/t/ann-datatoolkit-jl-reproducible-flexible-and-convenient-data-management/104757.md?page=1)