Happy to give a first look at a (as of yet unregistered) package
DataLoaders.jl that has a similar API to PyTorch’s
See examples in the
I wrote this primarily to support deep learning pipelines that load images, apply heavy preprocessing, and batch the samples. To do this without slowing down the training this has to be done on multiple threads and non-blockingly.
A few months ago I asked if something like this already exists in the Julia ecosystem, but did not find anything that suited my needs, so I decided to roll my own.
Note: in a recent release of
DataLoader was added, but as far as I can see, the implementation is neither parallel nor non-blocking.
I’d be happy about feedback, criticism, and whether you find the package helpful.