Extremely high memory consumption on CPU

[ToucheSir]

@cirobr

I wonder if BenchmarkTools is not considering GPU memory for the metric?

This is correct. To capture information about GPU allocations, check out the tools mentioned in Benchmarking & profiling · CUDA.jl.

@cirobr

For the CPU case, does the metric mean the model is too expensive for running on limited devices such as IoT application processors?

Maybe, maybe not. Equally if not more important than the total amount of memory allocated could be the maximum memory the model uses at any given point in time.

@cirobr

As a side question: its speed on gpu is 2-3X lower than equivalent models found elsewhere on GitHub. Have already applied many recommendations to reduce allocation, with little improvement. Any hint on where to look at for improvement tips is welcome.

There’s no central, definitive source, but plenty of previous discussions to search through with the obvious keywords. Just in the past little while on Discourse, I see Unreasonable memory usage with M4 GPU and Memory usage increasing with each epoch - #14 by JoshuaBillson. While on GitHub, cuda gpu memory usage increasing in time · Issue #2523 · FluxML/Flux.jl · GitHub covers much the same. There have been similar discussions on Slack.

Side note, are you aware of the existence of Machine Learning - Julia Programming Language ? Posts that are in too general a category can get lost, because some people only check specific categories frequently.