# Training a million models at the same time

**URL:** <https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [May 15, 2022, 2:41pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100 "2022-05-15T14:41:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![StefanPerko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefanperko/32/37647_2.png) [@StefanPerko](https://discourse.julialang.org/u/StefanPerko)\
**Post date:** [May 15, 2022, 2:41pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100/1 "2022-05-15T14:41:42Z")

</div>

I have written my own code for doing Monte-Carlo simulations with SGD and a one-parameter model, for research. Specifically, I would draw 10^6 data sets (x,y) from a known distribution with x,y 1-dimensional and train 10^6 instances of this model on the corresponding data sets by directly doing SGD on the a 10^6-element vector of parameters.

Can I do this in flux somehow too?

What I DON’T want is training each model separately. That would take me approximately 86 hours for a process that right now takes 10-30 seconds.

I also wanted to try training one model with a million parameters by using a custom loss function and treating the 10^6 as a dimension in the data. This doesn’t work either. It just tells me it runs out of memory if I only try to create the model with Dense(10^6 =\> 10^6, bias = false). The same happens for 10^5, but I takes maybe a minute before it tells me that. Maybe you could use 10^4-chunks, but that would still be much less efficient than my own code where I don’t have any issues with memory like this.

Any ideas whether there is a clever way of doing this already or do I need to still use my own code in the end?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [May 15, 2022, 4:14pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100/2 "2022-05-15T16:14:52Z")

</div>

> [@StefanPerko](#):
>
> It just tells me it runs out of memory if I only try to create the model with Dense(10^6 =\> 10^6, bias = false)

A dense layer connects each of the 10^6 inputs to each of the 10^6 outputs, which requires 10^12 weights, so it’s no surprise that you run out of memory. Presumably your own code is doing something different from a dense layer.

---

<div class="post-metadata">

**Author:** ![StefanPerko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefanperko/32/37647_2.png) [@StefanPerko](https://discourse.julialang.org/u/StefanPerko)\
**Post date:** [May 15, 2022, 5:06pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100/3 "2022-05-15T17:06:08Z")

</div>

Ah! Yes, of course. Sorry for the confusion. I need to force the weight matrix to be diagonal somehow - or I suppose I can probably just not use a Dense Layer and do it directly.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [May 15, 2022, 5:25pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100/4 "2022-05-15T17:25:08Z")

</div>

I’m not really sure I understand the training objective (sounds like a good candidate for using libraries from one of Julia’s PPL ecosystems), but if what you need is literally `Dense` with a diagonal weight matrix, we have [`Flux.Scale`](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Scale).

---

<div class="post-metadata">

**Author:** ![StefanPerko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefanperko/32/37647_2.png) [@StefanPerko](https://discourse.julialang.org/u/StefanPerko)\
**Post date:** [May 15, 2022, 6:12pm UTC](https://discourse.julialang.org/t/training-a-million-models-at-the-same-time/81100/5 "2022-05-15T18:12:04Z")

</div>

Perfect! That’s exactly what I need (because you can consider them 10^6 independent models).