# Integrating MLUtils.DataLoader and image augmentation pipeline on custom dataset

**URL:** <https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887>\
**Category:** Machine Learning\
**Created:** [February 1, 2023, 9:20pm UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887 "2023-02-01T21:20:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rkube/32/211198_2.png) [@rkube](https://discourse.julialang.org/u/rkube)\
**Post date:** [February 1, 2023, 9:20pm UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/1 "2023-02-01T21:20:43Z")

</div>

Hi,  
I’m trying to create a custom dataset where `getobs` performs random image augmentations.  
The [DataLoader](https://juliaml.github.io/MLUtils.jl/stable/api/#MLUtils.DataLoader) docs suggest that my dataset has to have custom `numobs` and `getobs` calls.

So this is my code:

```julia
struct my_dataset{T}
    data_arr::T
    pipeline::Augmentor.ImmutablePipeline
end

# Constructor sets the data and the pipeline
function my_dataset(
    data,
    pipeline)
    return my_dataset(data, pipeline)
end

function getobs(data::my_dataset, i)
    # Perform augmentations on single image. 
   return data_arr[i]
end

numobs(data::my_dataset) = size(data.data_arr)[end]

```

Instantiating a DataLoader works fine:

```julia
dset = my_dataset(randn(24, 8, 3, 10_000), Flipx() * NoOp())
loader = DataLoader(dset, batchsize=-1)

```

But when I try to iterate, instead of my `numobs` method, a [generic routine](https://github.com/JuliaML/MLUtils.jl/blob/ff2fcc1ba9e5690c0e393fe7e5003fcfabcd2d19/src/observation.jl#L53) is called:

```julia
julia> first(loader)
ERROR: MethodError: no method matching length(::my_dataset{Array{Float32, 4}})
Closest candidates are:
  length(::Union{Base.KeySet, Base.ValueIterator}) at abstractdict.jl:58
  length(::Union{LinearAlgebra.Adjoint{T, <:Union{StaticArraysCore.StaticArray{Tuple{var"#s2"}, T, 1} where var"#s2", StaticArraysCore.StaticArray{Tuple{var"#s3", var"#s4"}, T, 2} where {var"#s3", var"#s4"}}}, LinearAlgebra.Diagonal{T, <:StaticArraysCore.StaticArray{Tuple{var"#s13"}, T, 1} where var"#s13"}, LinearAlgebra.Hermitian{T, <:StaticArraysCore.StaticArray{Tuple{var"#s10", var"#s11"}, T, 2} where {var"#s10", var"#s11"}}, LinearAlgebra.LowerTriangular{T, <:StaticArraysCore.StaticArray{Tuple{var"#s18", var"#s19"}, T, 2} where {var"#s18", var"#s19"}}, LinearAlgebra.Symmetric{T, <:StaticArraysCore.StaticArray{Tuple{var"#s7", var"#s8"}, T, 2} where {var"#s7", var"#s8"}}, LinearAlgebra.Transpose{T, <:Union{StaticArraysCore.StaticArray{Tuple{var"#s2"}, T, 1} where var"#s2", StaticArraysCore.StaticArray{Tuple{var"#s3", var"#s4"}, T, 2} where {var"#s3", var"#s4"}}}, LinearAlgebra.UnitLowerTriangular{T, <:StaticArraysCore.StaticArray{Tuple{var"#s24", var"#s25"}, T, 2} where {var"#s24", var"#s25"}}, LinearAlgebra.UnitUpperTriangular{T, <:StaticArraysCore.StaticArray{Tuple{var"#s21", var"#s22"}, T, 2} where {var"#s21", var"#s22"}}, LinearAlgebra.UpperTriangular{T, <:StaticArraysCore.StaticArray{Tuple{var"#s15", var"#s16"}, T, 2} where {var"#s15", var"#s16"}}, StaticArraysCore.StaticArray{Tuple{var"#s25"}, T, 1} where var"#s25", StaticArraysCore.StaticArray{Tuple{var"#s1", var"#s3"}, T, 2} where {var"#s1", var"#s3"}, StaticArraysCore.StaticArray{<:Tuple, T}} where T) at ~/.julia/packages/StaticArrays/jA1zK/src/abstractarray.jl:1
  length(::Union{LinearAlgebra.Adjoint{T, S}, LinearAlgebra.Transpose{T, S}} where {T, S}) at ~/Software/julia-1.8.5/share/julia/stdlib/v1.8/LinearAlgebra/src/adjtrans.jl:172
  ...
Stacktrace:
 [1] numobs(::Type{SimpleTraits.Not{MLUtils.IsTable{kstar_ecei_dataset{Array{Float32, 4}}}}}, data::kstar_ecei_dataset{Array{Float32, 4}})
   @ MLUtils ~/.julia/packages/MLUtils/KcBtS/src/observation.jl:53
 [2] numobs
   @ ~/.julia/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [inlined]
 [3] ObsView(data::kstar_ecei_dataset{Array{Float32, 4}})
   @ MLUtils ~/.julia/packages/MLUtils/KcBtS/src/obsview.jl:145
 [4] iterate(e::DataLoader{kstar_ecei_dataset{Array{Float32, 4}}, Random._GLOBAL_RNG, Val{nothing}})
   @ MLUtils ~/.julia/packages/MLUtils/KcBtS/src/eachobs.jl:158
 [5] first(itr::DataLoader{kstar_ecei_dataset{Array{Float32, 4}}, Random._GLOBAL_RNG, Val{nothing}})
   @ Base ./abstractarray.jl:424
 [6] top-level scope
   @ REPL[40]:1

```

I don’t follow what is happening here. Why is my custom `numobs` method not called?

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [February 1, 2023, 9:26pm UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/2 "2023-02-01T21:26:32Z")

</div>

You likely need to import `numobs`

```julia
import MLUtils: numobs

```

or name it explicitly

```julia
MLUtils.numobs(data::my_dataset) = size(data.data_arr)[end]

```

By implementing it without importing or namespacing, you are defining a new function in the current module rather than adding a new method to the existing function.

---

<div class="post-metadata">

**Author:** ![rkube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rkube/32/211198_2.png) [@rkube](https://discourse.julialang.org/u/rkube)\
**Post date:** [February 1, 2023, 9:56pm UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/3 "2023-02-01T21:56:34Z")

</div>

Thanks, that works. Here is the MWE:

```julia
using MLUtils
using Random

struct my_dset{T}
    data_arr::T
    trf
end

function MLUtils.getobs(dset::my_dset, ix)
    obs = dset.data_arr[:, ix]
    map(dset.trf, obs)
end
MLUtils.numobs(data::my_dset) = size(data.data_arr)[end]

d = randn(Float32, 3, 20)
ds = my_dset(d, x -> x + 12.3)

loader = DataLoader(ds, batchsize=-1)

for obs ∈ loader
    @show obs
end

```

---

<div class="post-metadata">

**Author:** ![rkube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rkube/32/211198_2.png) [@rkube](https://discourse.julialang.org/u/rkube)\
**Post date:** [February 1, 2023, 10:01pm UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/4 "2023-02-01T22:01:07Z")

</div>

As a side-note, this approach would be the julia-prototype for [pytorch-like dataloaders](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html). Is there a tutorial like this but for julia anywhere on the web?

---

<div class="post-metadata">

**Author:** ![jeremiedb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiedb/32/29150_2.png) [@jeremiedb](https://discourse.julialang.org/u/jeremiedb)\
**Post date:** [February 2, 2023, 3:07am UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/5 "2023-02-02T03:07:44Z")

</div>

For reference when dealing with larger image datasets requiring to load them from disk, the following can be used: [ImageNetTrain.jl/resnet-optim.jl at b7cc19676a74525d9b4ec007435f2ff9c892c604 · jeremiedb/ImageNetTrain.jl · GitHub](https://github.com/jeremiedb/ImageNetTrain.jl/blob/b7cc19676a74525d9b4ec007435f2ff9c892c604/resnet-optim.jl#L127-L150)

Note that it is sufficient to extend `Base`’s `length` and `getobs` to get a working custom dataloader (`MLUtils`’s `numobs` not required)

---

<div class="post-metadata">

**Author:** ![CarloLucibello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carlolucibello/32/3278_2.png) [@CarloLucibello](https://discourse.julialang.org/u/CarloLucibello)\
**Post date:** [February 2, 2023, 7:05am UTC](https://discourse.julialang.org/t/integrating-mlutils-dataloader-and-image-augmentation-pipeline-on-custom-dataset/93887/6 "2023-02-02T07:05:08Z")

</div>

`getobs` and `numbobs` fallback to `getindex` and `length` so the following works as well:

```julia
struct my_dset{T}
    data_arr::T
    trf
end

function Base.getindex(dset::my_dset, ix)
    obs = dset.data_arr[:, ix]
    map(dset.trf, obs)
end

Base.length(data::my_dset) = size(data.data_arr)[end]

```