[ANN] DataToolkit.jl — Reproducible, flexible, and convenient data management

Package Announcements
, , ,
21

I’m going to put on a burst of effort to get v0.10 out the door, leaning on LLMs to accelerate the testing/bugfixing/documenting process.

I faintly recall some ideas for further features/enhancements, but at the moment the changelog is pretty much what I shared earlier. Restated for convenience:

DataToolkit v0.10

Highlights

  • Directory support: datasets can now be whole directories, not just single files, with data integrity preserved via cached Merkle-tree checksumming.
  • Package restructuring: DataToolkitBase was renamed to DataToolkitCore, and a new lighter-weight DataToolkitBase was introduced for package authors (data dependencies without the data> REPL). The user-facing DataToolkit retains the full REPL experience.
  • Monorepo: all packages (DataToolkit, DataToolkitBase, DataToolkitCore, DataToolkitStore, DataToolkitREPL, DataToolkitCommon) now live in a single DataToolkit.jl repository.

New features

  • Open files as a FilePathsBase.AbstractPath.
  • Basic S3 download support.
  • More image formats: GIF and WebP.
  • REPL mode and the content store are now split into their own packages (DataToolkitREPL, DataToolkitStore).

Improvements

  • Faster load times and precompilation.
  • Better logging: moved from DataToolkitCommon into DataToolkitCore, reworked, and now configured via Preferences.
  • Improved type inference and dispatch logic.
  • A lot of bugfixes
  • Documentation improvements (ongoing).

Under the hood

  • Minimum Julia version raised to 1.9, adopting package extensions.
  • @import replaced with @require for lazy package loading.
  • Removed the internal SmallDict type (Julia 1.11’s Memory addresses the original motivation).

If there are any enhcancements you’d like to see, now is an excellent time to mention them :slight_smile:

22

Congrats! ps. some of the links in the original post are dead. Including the showcase video :slight_smile:

23

Thanks for the work on DataToolkit! If now’s the time to share a wishlist, I’d be amazing to be able to pull dependencies from a Google Doc folder, maybe prompting the user for some authentication when needed. Maybe this can be achieved via rclone integration?

24

It is a good time to share a wishlist!

For pulling from google drive, the two questions that would help are:

  • Is there a good way to fetch from google drive in Julia?
  • What’s a good way to stash credentials? Some sort of cross-platform keyring package seems like what we ideally want.
25

Unfortunately GitHub - alecloudenback/Keyring.jl · GitHub doesn’t look particularly lively

26

I know that GoogleDrive.jl exists, but I also remember some pain points in using it because I have never been able to make it download a complete folder, and the URLs are hard to guess, in opposition to rclone where you can have it download a folder based on its path within your own drive.

It also seems that someone made an interface to rclone in Julia, but I haven’t tried it: GitHub - imohag9/RcloneInterface.jl: Julia interface to Rclone a command-line program to manage files on cloud storage · GitHub.

27

There are multiple services using authentication I would like to support, I just want to handle them well, and not in a “dump a secret into a plain file” type way.

Rclone seems like a good thing to look at interfacing with, and worst case I might end up having a stab at finishing Keyring.jl (for some future version).

28

Oh yes, of course. Unfortunately I have zero knowledge on that part…

29

I can’t say I have much either, but I’ve never let that stop me before :joy: