Join / Append dataLoaders structures

Hi all,

How can I add / join /append two dataLoaders structures ?

I’m not aware of a way to do this, as the general advice is to join your data(sets) before creating a loader. Is there anything which would prevent you from doing that instead?


The objective is to “balance” the training dataset. I have an unbalanced dataset (80%- 20%) and I want to create a kind of upsample balanced mini-batches for training ensuring that in each mini-batches aren’t repeated data.
If I upsample and after creating the (shuffle) data mini-batches the last condition is not ensured. Although I’m new to Julia, I have done a quick look at the source code of the dataloader, and seems that is easier to try modifying the FLUX training loop to have into account multiple dataloaders objects .

I know the type of sampling you’re talking about, thanks for the clarification. Since you have a good grasp on what the training loop should look like, I’d recommend just writing a custom one. Flux’s default is very constrained and this is one of those cases where the intent of the balancing might be clearer if the iteration over multiple dataloaders is made explicit.