# Allocation of Memory while evaluate a model

**URL:** <https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [November 6, 2021, 8:52pm UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064 "2021-11-06T20:52:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![HenrZu](https://avatars.discourse-cdn.com/v4/letter/h/3ab097/32.png) [@HenrZu](https://discourse.julialang.org/u/HenrZu)\
**Post date:** [November 6, 2021, 8:52pm UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/1 "2021-11-06T20:52:59Z")

</div>

While working on the performance of a model used in a numerical simulation, I noticed that memory is allocated every time a model is evaluated.

Since I call the model several times within the simulation, there is a significant loss in runtime.

Already with a simple model with an input of length 10 and a binary classification as output, more than 800 bytes are allocated.

```julia
using Flux
using BenchmarkTools

model = Chain(
  Dense(10, 5, σ),
  Dense(5, 2),
  softmax)

input = Vector{Float64}(rand(10))
@benchmark model($input)

```

```julia
BenchmarkTools.Trial: 10000 samples with 187 evaluations.
 Range (min … max): 539.037 ns … 47.713 μs ┊ GC (min … max): 0.00% … 98.36%
 Time (median): 568.449 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 656.073 ns ± 1.633 μs ┊ GC (mean ± σ): 9.27% ± 3.67%

    ▁█▄▄▇▂                  
  ▁▂██████▅▃▃▂▂▂▁▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  539 ns Histogram: frequency by time 824 ns <

 Memory estimate: 848 bytes, allocs estimate: 10.

```

Is there a way to evaluate the model without allocating memory?

CC: @sloede

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [November 7, 2021, 3:51am UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/2 "2021-11-07T03:51:29Z")

</div>

If your networks are small and allocations really are the bottleneck, you could take a look at [SimpleChains.jl](https://juliahub.com/ui/Packages/SimpleChains/If4q9/0.1.10). I believe it’s still experimental, but it allows for preallocating certain buffers whereas Flux does pretty much everything out-of-place (mostly for AD compat).

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [November 7, 2021, 11:39am UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/3 "2021-11-07T11:39:40Z")

</div>

Thanks for the suggested package - it seems like it maybe could do the trick. However, the referenced repo does not seem to exist anymore - at least the GH URL [https://github.com/JuliaSIMD/SimpleChains.jl](https://github.com/JuliaSIMD/SimpleChains.jl) gives me a 404.

---

<div class="post-metadata">

**Author:** ![HenrZu](https://avatars.discourse-cdn.com/v4/letter/h/3ab097/32.png) [@HenrZu](https://discourse.julialang.org/u/HenrZu)\
**Post date:** [November 7, 2021, 5:23pm UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/4 "2021-11-07T17:23:48Z")

</div>

Thanks a lot for the answer. SimpleChains looks really interesting and could be a possible solution to my problem.

Unfortunately, as Michael already mentioned, the repo returns a 404.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [November 7, 2021, 7:17pm UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/5 "2021-11-07T19:17:04Z")

</div>

IIRC the repo has been made private, but you can still use the registered package.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [November 8, 2021, 9:40am UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/6 "2021-11-08T09:40:55Z")

</div>

@Elrod Would you mind shedding some light on the whereabouts of SimpleChains.jl and your future plans for it?

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [November 8, 2021, 10:52am UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/7 "2021-11-08T10:52:59Z")

</div>

As ToucheSir noted, it’s been made private.  
When/if it can be opened again will require an internal discussion/ business plan delineating how/where it fits.  
It almost certainly won’t be a product itself, but may be a piece of some other offering, so I think we should open it.  
I jumped the gun by creating it as an open repo in the first place…

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [November 8, 2021, 11:02am UTC](https://discourse.julialang.org/t/allocation-of-memory-while-evaluate-a-model/71064/8 "2021-11-08T11:02:05Z")

</div>

Thanks for the clarification. Thus, to conclude, there does not seem to be a “canonical” Julia package for ANN models available at the moment that does not allocate during each model evaluation. That’s too bad, and I would like to - in case anyone cares 😉 - express strong interest in having such an option available for Julia!