# PyTorch DataLoader equivalent for training large models with Flux

**URL:** <https://discourse.julialang.org/t/pytorch-dataloader-equivalent-for-training-large-models-with-flux/30763>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [November 5, 2019, 6:25pm UTC](https://discourse.julialang.org/t/pytorch-dataloader-equivalent-for-training-large-models-with-flux/30763 "2019-11-05T18:25:02Z")\
**Posts on this page:** 1\
**Showing post:** 17

<div class="post-metadata">

**Author:** ![terasakisatoshi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/terasakisatoshi/32/9187_2.png) [@terasakisatoshi](https://discourse.julialang.org/u/terasakisatoshi)\
**Post date:** [November 8, 2020, 8:58am UTC](https://discourse.julialang.org/t/pytorch-dataloader-equivalent-for-training-large-models-with-flux/30763/17 "2020-11-08T08:58:03Z")

</div>

How about [holylorenzo](https://discourse.julialang.org/u/holylorenzo)’s repository?

> [@\[ANN\] DataLoaders.jl (alpha) - basically PyTorch's parallel \`DataLoader\`](https://discourse.julialang.org/t/ann-dataloaders-jl-alpha-basically-pytorchs-parallel-dataloader/35963):
>
> Happy to give a first look at a (as of yet unregistered) package [DataLoaders.jl](https://github.com/lorenzoh/DataLoaders.jl) that has a similar API to PyTorch’s DataLoader. See examples in the [README.md](https://github.com/lorenzoh/DataLoaders.jl/blob/master/README.md) I wrote this primarily to support deep learning pipelines that load images, apply heavy preprocessing, and batch the samples. To do this without slowing down the training this has to be done on multiple threads and non-blockingly. A few months ago [I asked](https://discourse.julialang.org/t/pytorch-dataloader-equivalent-for-training-large-models-with-flux/30763/14) if something like this already exists in the Julia ecosystem, but did not find anyt…

---

_[View the full topic](https://discourse.julialang.org/t/pytorch-dataloader-equivalent-for-training-large-models-with-flux/30763)._