# How to handle dataloader consisting of data with different dimensions?

**URL:** <https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [August 23, 2021, 2:17pm UTC](https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847 "2021-08-23T14:17:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kengyangyao](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@kengyangyao](https://discourse.julialang.org/u/kengyangyao)\
**Post date:** [August 23, 2021, 2:17pm UTC](https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847/1 "2021-08-23T14:17:25Z")

</div>

Hi, I was trying to use `Flux.DataLoader` with following code:

```julia
X1 = rand(85761, 8, 24); X2 = rand(85761, 8, 1); y_1 = rand(85761, 8, 1); emb = rand(85761, 1)
train_loader = Flux.DataLoader((X1=X1, emb=emb, X2=X2, y_1=y_1, ), batchsize=500, shuffle=true);

```

but I got following warning:

`DimensionMismatch("All data inputs should have the same number of observations, i.e. size in the last dimension. But data[:X1] (85761×8×24 Array{Float64, 3}) has 24, while data[:emb] (85761×1 Matrix{Float64}) has 1.") `  
I wonder why the number of data points is determined by last dimension instead of first dimension? And how should I deal with my data in this case (swap dimension 1 and 3, or rotate dimension (1,2,3) → (2, 3,1)?

Thank you!

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [August 23, 2021, 5:00pm UTC](https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847/2 "2021-08-23T17:00:06Z")

</div>

> [@kengyangyao](#):
>
> I wonder why the number of data points is determined by last dimension instead of first dimension?

Afaik, this has to do with julia being column major instead of row major.

As for how to handle, it depends on what you want to feed the data into. The docs for each layertype in flux should describe what they assume about the data.

---

<div class="post-metadata">

**Author:** ![DiracFermion1](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@DiracFermion1](https://discourse.julialang.org/u/DiracFermion1)\
**Post date:** [August 23, 2021, 5:23pm UTC](https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847/4 "2021-08-23T17:23:11Z")

</div>

What does “column major” mean here? Does it mean “number of data points” should be the last dimension rather than first dimension?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 23, 2021, 5:50pm UTC](https://discourse.julialang.org/t/how-to-handle-dataloader-consisting-of-data-with-different-dimensions/66847/5 "2021-08-23T17:50:41Z")

</div>

“column major” means that the first dimension during indexing iterates over columns of multidimensional data first. As for `Flux.DataLoader`, it was decided that since the first thing to be iterated over is a column, it makes sense to have batches be stored in columns, since that way the thing you iterate first over is a single batch.