# DataLoaders.jl Workers systematically end up outside bounds
**URL:** https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928
**Category:** Machine Learning
**Tags:** question
**Created:** [June 15, 2021, 7:10am UTC](https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928 "2021-06-15T07:10:16Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@ClarkGuilty](https://discourse.julialang.org/u/ClarkGuilty)
#### Post date: [June 15, 2021, 7:10am UTC](https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928/1 "2021-06-15T07:10:16Z")
Hello fellow Julians. I am trying to train a model using a parallel DataLoader, however, following the example from the documentation (available here: [https://github.com/lorenzoh/DataLoaders.jl/blob/master/docs/datacontainers.md](https://github.com/lorenzoh/DataLoaders.jl/blob/master/docs/datacontainers.md)) fails.
The general idea is training a model but only loading data to memory before using it, since loading the entire dataset requieres more RAM than the one available.
When iterating using the dataloader, the workers systematically end up outside the bounds defined by nobs() and the @async never really finishes but waits eternally for the failed workers.
I have a MWE available here [https://github.com/ClarkGuilty/PreliminaryJuliaTests/blob/master/DataLoading.jl](https://github.com/ClarkGuilty/PreliminaryJuliaTests/blob/master/DataLoading.jl).
Are you successfully using DataLoaders.jl without any modification on the source?
---
### Author:  [@holylorenzo](https://discourse.julialang.org/u/holylorenzo)
#### Post date: [June 15, 2021, 9:07am UTC](https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928/2 "2021-06-15T09:07:41Z")
Hi Clark, author here. The issue seems to be a discrepancy in how `MLDataPattern.BatchView` and `DataLoaders.BatchViewCollated` interpret `nobs`. For the latter, every batch is one observation, but the former treats counts every single observation.
For your use case, passing `collate = true` to the `DataLoader` call should fix it; this is probably what you want anyway if you’re training on batches. Let me know if you run into any other problems. I’ll raise the inconsistency as an issue.
---
### Author:  [@ClarkGuilty](https://discourse.julialang.org/u/ClarkGuilty)
#### Post date: [June 18, 2021, 12:10am UTC](https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928/3 "2021-06-18T00:10:46Z")
Hi HolyLorenzo, thank you for your answer, and your package is a blessing. Regarding your suggestion, I changed it to `collate = true` but had an error relating to the copyrec!
```julia
Base.StackTraces.StackFrame[(::DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}})(args::Int64) at workerpool.jl:55, macro expansion at workerpool.jl:69 [inlined], #8 at macros.jl:19 [inlined], #62 at qpool.jl:195 [inlined], (::ThreadPools.var"#58#59"{ThreadPools.var"#62#64"{DataLoaders.var"#8#14"{DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}}}}, Tuple{Int64, Int64}})() at qpool.jl:86]
┌ Error: Exception while executing task on worker 3. Shutting down WorkerPool.
│ e =
│ MethodError: no method matching copyrec!(::SubArray{Int64, 0, Vector{Int64}, Tuple{Int64}, true}, ::Int64)
│ Closest candidates are:
│ copyrec!(::AbstractArray, ::AbstractArray) at /home/hipparcos/.julia/dev/DataLoaders/src/batchview.jl:112
│ stacktrace = 6-element Vector{Base.StackTraces.StackFrame}: …
│ args = 1
└ @ DataLoaders ~/.julia/dev/DataLoaders/src/workerpool.jl:56
^C
ERROR: LoadError: InterruptException:
Stacktrace:
[1] take_buffered(c::Channel{Tuple{Matrix{Float32}, Vector{Int64}}})
@ Base ./channels.jl:389
[2] take!
@ ./channels.jl:383 [inlined]
[3] take!(ringbuffer::DataLoaders.RingBuffer{Tuple{Matrix{Float32}, Vector{Int64}}})
@ DataLoaders ~/.julia/dev/DataLoaders/src/ringbuffer.jl:55
[4] iterate
@ ~/.julia/dev/DataLoaders/src/loaders.jl:103 [inlined]
[5] iterate(iterparallel::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}})
@ DataLoaders ~/.julia/dev/DataLoaders/src/loaders.jl:89
[6] macro expansion
@ ~/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
[7] train!(loss::Function, ps::Zygote.Params, data::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}}, opt::Descent; cb::Flux.Optimise.var"#40#46")
@ Flux.Optimise ~/.julia/packages/Flux/0c9kI/src/optimise/train.jl:99
[8] train!(loss::Function, ps::Zygote.Params, data::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}}, opt::Descent)
@ Flux.Optimise ~/.julia/packages/Flux/0c9kI/src/optimise/train.jl:97
[9] top-level scope
@ ~/pre_PhD/phd_pretesis/PreliminaryJuliaTests/DataLoading.jl:67
in expression starting at /home/hipparcos/pre_PhD/phd_pretesis/PreliminaryJuliaTests/DataLoading.jl:67
```
After that, I noticed that copyrec! is only used when there is no getobs!, so I implemented it:
```julia
function LearnBase.getobs!(buf, dataset::ImageDataset, i::Int) :: (Vector{Float32}, Int64)
subpath = dataset.files[i]
file = joinpath(dataset.dir, subpath)
buf[1] = reshape(AstroImage(file).data[1],101*101)
buf[2] = dataset.classes[i]
buf
end
function LearnBase.getobs!(buf, dataset::ImageDataset, range::UnitRange{Int64}) :: (Vector{Float32}, Int64)
subpath = dataset.files[range]
file = joinpath.(dataset.dir, subpath)
buf[1] = hcat(give_data.(AstroImage.(file))...)
buf[2] = dataset.classes[range]
buf
end
```
but then I got:
```julia
Base.StackTraces.StackFrame[(::DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}})(args::Int64) at workerpool.jl:55, macro expansion at workerpool.jl:69 [inlined], #8 at macros.jl:19 [inlined], #62 at qpool.jl:195 [inlined], (::ThreadPools.var"#58#59"{ThreadPools.var"#62#64"{DataLoaders.var"#8#14"{DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}}}}, Tuple{Int64, Int64}})() at qpool.jl:86]
Base.StackTraces.StackFrame[(::DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}})(args::Int64) at workerpool.jl:55, macro expansion at workerpool.jl:69 [inlined], #8 at macros.jl:19 [inlined], #62 at qpool.jl:195 [inlined], (::ThreadPools.var"#58#59"{ThreadPools.var"#62#64"{DataLoaders.var"#8#14"{DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}}}}, Tuple{Int64, Int64}})() at qpool.jl:86]
┌ Error: Exception while executing task on worker 2. Shutting down WorkerPool.
│ e = MethodError: no method matching setindex!(::Tuple{SubArray{Float32, 1, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, Int64}, true}, SubArray{Int64, 0, Vector{Int64}, Tuple{Int64}, true}}, ::Vector{Float32}, ::Int64)
│ stacktrace = 6-element Vector{Base.StackTraces.StackFrame}: …
│ args = 1
└ @ DataLoaders ~/.julia/dev/DataLoaders/src/workerpool.jl:56
Base.StackTraces.StackFrame[(::DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}})(args::Int64) at workerpool.jl:55, macro expansion at workerpool.jl:69 [inlined], #8 at macros.jl:19 [inlined], #62 at qpool.jl:195 [inlined], (::ThreadPools.var"#58#59"{ThreadPools.var"#62#64"{DataLoaders.var"#8#14"{DataLoaders.var"#inloop#10"{DataLoaders.WorkerPool{Int64}}}}, Tuple{Int64, Int64}})() at qpool.jl:86]
┌ Error: Exception while executing task on worker 4. Shutting down WorkerPool.
│ e = MethodError: no method matching setindex!(::Tuple{SubArray{Float32, 1, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, Int64}, true}, SubArray{Int64, 0, Vector{Int64}, Tuple{Int64}, true}}, ::Vector{Float32}, ::Int64)
│ stacktrace = 6-element Vector{Base.StackTraces.StackFrame}: …
│ args = 2
└ @ DataLoaders ~/.julia/dev/DataLoaders/src/workerpool.jl:56
┌ Error: Exception while executing task on worker 3. Shutting down WorkerPool.
│ e = MethodError: no method matching setindex!(::Tuple{SubArray{Float32, 1, Matrix{Float32}, Tuple{Base.Slice{Base.OneTo{Int64}}, Int64}, true}, SubArray{Int64, 0, Vector{Int64}, Tuple{Int64}, true}}, ::Vector{Float32}, ::Int64)
│ stacktrace = 6-element Vector{Base.StackTraces.StackFrame}: …
│ args = 3
└ @ DataLoaders ~/.julia/dev/DataLoaders/src/workerpool.jl:56
^C
ERROR: LoadError: InterruptException:
Stacktrace:
[1] take_buffered(c::Channel{Tuple{Matrix{Float32}, Vector{Int64}}})
@ Base ./channels.jl:389
[2] take!
@ ./channels.jl:383 [inlined]
[3] take!(ringbuffer::DataLoaders.RingBuffer{Tuple{Matrix{Float32}, Vector{Int64}}})
@ DataLoaders ~/.julia/packages/DataLoaders/uGlPg/src/ringbuffer.jl:55
[4] iterate
@ ~/.julia/dev/DataLoaders/src/loaders.jl:103 [inlined]
[5] iterate(iterparallel::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}})
@ DataLoaders ~/.julia/packages/DataLoaders/uGlPg/src/loaders.jl:89
[6] macro expansion
@ ~/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
[7] train!(loss::Function, ps::Zygote.Params, data::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}}, opt::Descent; cb::Flux.Optimise.var"#40#46")
@ Flux.Optimise ~/.julia/packages/Flux/0c9kI/src/optimise/train.jl:99
[8] train!(loss::Function, ps::Zygote.Params, data::DataLoaders.BufferGetObsParallel{Tuple{Matrix{Float32}, Vector{Int64}}, DataLoaders.BatchViewCollated{ImageDataset}}, opt::Descent)
@ Flux.Optimise ~/.julia/packages/Flux/0c9kI/src/optimise/train.jl:97
[9] top-level scope
@ ~/pre_PhD/phd_pretesis/PreliminaryJuliaTests/DataLoading.jl:84
in expression starting at /home/hipparcos/pre_PhD/phd_pretesis/PreliminaryJuliaTests/DataLoading.jl:84
```
There is a MWE now with `collate = true` and implementations of getobs! in the same repository.
---
### Author:  [@Alexander-Barth](https://discourse.julialang.org/u/Alexander-Barth)
#### Post date: [September 22, 2021, 9:55am UTC](https://discourse.julialang.org/t/dataloaders-jl-workers-systematically-end-up-outside-bounds/62928/4 "2021-09-22T09:55:35Z")
I had the same issue, but `collate = true` solved the problem for me. Thanks @holylorenzo !