Unfortunately the dataset does not fit into memory.
Dataloader doesn’t do lazy-loading, it’s
Dataset that does.
Although it’s not as convenient as
Dataset, you can implement a function based on MappedArrays.jl, for example:
files = map(x->joinpath(root, x), readdir(root))
return mappedarray(load, files)
root = "/Users/jc/Downloads/dataset"
# add ; in interactive environment to disable loading all files
dataset = load_dataset(root);
Image is not read from disk until it’s used, and that’s the trick
Dataset does to save your memory. However, the same image is read from disk multiple times, which would be slower than directly reading from memory. It’s a tradeoff between computational time and memory space.