I’m trying to train a large computer vision model I built in Flux. Unfortunately the dataset does not fit into memory. Does something similar to PyTorch’s DataLoader exist as a Julia package? Is there an ongoing effort on creating one? Has anyone else come across the need for something like it befo…

PyTorch DataLoader equivalent for training large models with Flux

baggepinnen November 27, 2019, 9:30am 16

2 Likes

Topic		Replies	Views
[ANN] DataLoaders.jl (alpha) - basically PyTorch's parallel `DataLoader` Package Announcements flux , machine-learning	0	879	March 14, 2020
Training FLUX models with larger datasets Machine Learning cuda , flux	4	1376	April 7, 2022
Parallel data loading to GPU arrays Machine Learning gpu , parallel , gpuarrays , data , flux	3	1099	January 30, 2019
[ANN] LengthChannels - Buffered iterators for machine learning Package Announcements package , announcement	2	727	November 28, 2019
How to use dataloader New to Julia flux	0	281	October 31, 2020