Allocation of Memory while evaluate a model

Specific Domains Machine Learning
1

While working on the performance of a model used in a numerical simulation, I noticed that memory is allocated every time a model is evaluated.

Since I call the model several times within the simulation, there is a significant loss in runtime.

Already with a simple model with an input of length 10 and a binary classification as output, more than 800 bytes are allocated.

using Flux
using BenchmarkTools

model = Chain(
  Dense(10, 5, Οƒ),
  Dense(5, 2),
  softmax)

input = Vector{Float64}(rand(10))
@benchmark model($input)

BenchmarkTools.Trial: 10000 samples with 187 evaluations.
 Range (min … max):  539.037 ns … 47.713 ΞΌs  β”Š GC (min … max): 0.00% … 98.36%
 Time  (median):     568.449 ns              β”Š GC (median):    0.00%
 Time  (mean Β± Οƒ):   656.073 ns Β±  1.633 ΞΌs  β”Š GC (mean Β± Οƒ):  9.27% Β±  3.67%

    β–β–ˆβ–„β–„β–‡β–‚                  
  β–β–‚β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–…β–ƒβ–ƒβ–‚β–‚β–‚β–β–‚β–‚β–‚β–‚β–‚β–‚β–‚β–β–β–β–β–β–β–β–β–β–β–β–β–β–β–β–β–β–‚β–‚β–ƒβ–‚β–‚β–‚β–‚β–‚β–β–β–β–β–β–β–β–β–β–β–β–β– β–‚
  539 ns          Histogram: frequency by time          824 ns <

 Memory estimate: 848 bytes, allocs estimate: 10.

Is there a way to evaluate the model without allocating memory?

CC: @sloede

1 Like
2

If your networks are small and allocations really are the bottleneck, you could take a look at SimpleChains.jl. I believe it’s still experimental, but it allows for preallocating certain buffers whereas Flux does pretty much everything out-of-place (mostly for AD compat).

2 Likes
3

Thanks for the suggested package - it seems like it maybe could do the trick. However, the referenced repo does not seem to exist anymore - at least the GH URL https://github.com/JuliaSIMD/SimpleChains.jl gives me a 404.

1 Like
4

Thanks a lot for the answer. SimpleChains looks really interesting and could be a possible solution to my problem.

Unfortunately, as Michael already mentioned, the repo returns a 404.

5

IIRC the repo has been made private, but you can still use the registered package.

1 Like
6

@Elrod Would you mind shedding some light on the whereabouts of SimpleChains.jl and your future plans for it?

1 Like
7

As ToucheSir noted, it’s been made private.
When/if it can be opened again will require an internal discussion/ business plan delineating how/where it fits.
It almost certainly won’t be a product itself, but may be a piece of some other offering, so I think we should open it.
I jumped the gun by creating it as an open repo in the first place…

1 Like
8

Thanks for the clarification. Thus, to conclude, there does not seem to be a β€œcanonical” Julia package for ANN models available at the moment that does not allocate during each model evaluation. That’s too bad, and I would like to - in case anyone cares :wink: - express strong interest in having such an option available for Julia!

1 Like