Virtual (or lazy) representation of a repeated array

Specific Domains Machine Learning
,
4

This seems like a bit of an XY problem. Why create this virtual array at all? You seem to want to circumvent the Flux.DataLoader, and return the same data each time. Why not just make that array once, something like x1 = repeat(tmp, 1, sz[2], sz[3], batchsize), and use it every time, instead of pulling this from the DataLoader.

That said I am a bit surprised that LazyArryas fails, but there are probably dozens of other packages which would let you create such a virtual array. Here’s one, with a smaller example.

julia> tmp = rand(Int8, 4)
4-element Vector{Int8}:
 109
  92
  72
  90

julia> x = repeat(tmp, 1, 5)
4×5 Matrix{Int8}:
 109  109  109  109  109
  92   92   92   92   92
  72   72   72   72   72
  90   90   90   90   90

julia> using LazyStack

julia> lazystack(fill(tmp, 5))
4×5 lazystack(::Vector{Vector{Int8}}) with eltype Int8:
 109  109  109  109  109
  92   92   92   92   92
  72   72   72   72   72
  90   90   90   90   90

julia> using Flux

julia> Flux.DataLoader(lazystack(fill(tmp, 5)))
5-element DataLoader(lazystack(::Vector{Vector{Int8}}))
  with first element:
  4×1 Matrix{Int8}

julia> first(ans)
4×1 Matrix{Int8}:
 109
  92
  72
  90
3 Likes